What is the First Directory Crawler?

First Directory are a leading provider of UK industrial business information. To keep our information as up to date as possible we regularly crawl companies sites looking for updated information and to get relevant screen shots.


How often will my site get visited by the First Directory Crawler?

The update-intervals depend on both the size of your website as well as its popularity. We will try our best to make sure we do not put any excess strain on your web servers. This means that usually only one crawler will search through your domain's content at a time. Should we notice that response-times are starting to lag, we will finish our crawling process at a more convenient time.


Will the First Directory Crawler obey the content of my robots.txt?

The First Directory Crawler will obey the Robot Exclusion Standard and will take the entries in your robots-txt file into account. To add exceptions for just the First Directory Crawler, please use 'firstdirectory-bot' for the user-agent-specification.


Do you have any other questions? Please send them to crawler@1stdirectory.net and we will try to get back to you as soon as possible.