If you are reading this page, it is likely that your web site was crawled by our search engine spider. Named RoboCrawl, our spider is smart in a way it can recognise many commands teaching it not to touch certain directories, or completely skip your site. We assure you RoboCrawl is a friendly robot and it follows the Robot Exclusion Standard.

Please read on if you wish to find out how to control RoboCrawl as it spiders your site.

Robot Exclusion Standard

If you do noy want your web page spidered by our crawler, you can simply create a robots.txt file in your root web directory with specific variables as found at the Robot Exclusion Standard web site. You can prevent Canadian Content from crawling your whole site, and never return again or you can specify certain directories in which you would like crawled and not to be crawled.

Control RoboCrawl with metatags

If you still want your site indexed, but want to make some pages off-limits, it's very simple to do so. Simply add a tag in your html "<META NAME="robots" CONTENT="NOINDEX">", more information about these meta tags can be found at Authors Guide to META Robots Tag.

Time between visits

RoboCrawl will only visit your everytime we re-index, which is approximately every 30 days. You can specify how long before your site is recrawled with the "revisit" metatag. However, when and if a web site is caught spamming, ie. using hidden keywords, submitting a page too often or having over two popup windows. The URL will be banned from our index and the crawler will never visit such sites in the future.

How long will it take to appear in the index?

The time it takes for your URL to show up in our index can take anywhere from one day after it has been crawled to one month after it has been crawled. Don't be alarmed if your url does not appear in our index a while after it has been crawled, we might just be dealing with a large amount of urls which have to be properly indexed.

What is a snapshot?

Canadian Content will take a snapshot of the page when it crawled the site allowing users to view the site exactly as it appear when it was last indexed. Cached snapshots of pages usually rotate every month or so. However, large pages will not be fully indexed, and some HTML might be found missing on a snapshot if a page is too large.

How can I submit my site?

After you have submitted your site at Canadian Content suggest a site, it can take anywhere from 1 days to a matter of weeks till our spider gets around to crawling your site. Don't be worried if your site isn't crawled within a week, our spider just hasn't finished previously submitted sites.