Sqwidge.com is what we consider an exciting new search engine currently running in alpha development stage. Our primary objective when searching is to unearth great content while making sure that we abide by all your rule sets for your site. Please bear with us while we constantly develop and improve our spidering methods.
Below we attempt to answer your questions about the SqwidgeBot crawler so you can gain more insight into what it's looking for.
SqwidgeBot works in an automated manner, it crawls your site looking at the content to add to our Index. Currently we are running in Alpha development so the results aren't published to the general public just yet ... but soon!
That is perfectly fine, we don't want to upset anybody if possible and we obey the Robot Exclusion standard. To prevent a file or directory of your site from being crawled, add a file named robots.txt in the root directory (example: http://www.domain.com/robots.txt). In this file you are able to specify disallowed paths of your website in the following format:
User-agent: SqwidgeBot
Disallow: /private
Disallow: /everything
In the above example we have told SqwidgeBot not to access any files located in the directory named /private and /everything.
For more information on Robot Exclusion please visit www.robotstxt.org
Do not worry... We also follow another popular standard; by inserting a Meta Tag in the pages you want to disallow we will listen and make sure we don't index.
Example of the Meta Tag mentioned is as follows:
<Meta name="SqwidgeBot" content="noindex">
If you don't want SqwidgeBot to follow any links on a page simply add the following meta tag:
<Meta name="SqwidgeBot" content="nofollow">
The two restrictions can be written using one tag like so:
<Meta name="SqwidgeBot" content="noindex,nofollow">
Please note though we will have to look at the page first to discover that you do not wish us to index the page.
Definitely! Sometimes the speeds at which SqwidgeBot browses website's can be a little bit too fast for you so we have made every effort to support the Crawl-Delay robot directive.
User-agent: SqwidgeBot
Crawl-Delay: 5
The robots.txt example above will limit the downloading of a page to every Ten seconds. You can change this number to however many seconds you like. Remember to always keep it as seconds though.
As part of our quality control effort we recognize automatically multiple domains on the same IP and limit the number of requests made to the IP.
In short, Yes, of course! We make every effort to index as many web page's as possible. But given the ever changing nature of a dynamic page we limit the number of pages we download during a visit.
