Example Robots.txt Format

Allow indexing of everything
User-agent: *
Disallow:

Disallow indexing of everything
User-agent: *
Disallow: /

Disallow Googlebot from indexing of a folder, except for allowing the indexing of one file in that folder
User-agent: Googlebot
Disallow: /folder1/
Allow: /folder1/myfile.html

You can set Yahoo! Slurp crawl delays in your robots.txt file. Yahoo! offers background information

Their robots.txt crawl delay code looks like
User-agent: Slurp
Crawl-delay: 5
where the 5 is in seconds.


Microsoft's information is located in their Live Help menu , but is a bit harder to find. Their robots.txt crawl delay code looks like
User-agent: msnbot
Crawl-delay: 10
where the 10 is in second