Robots.txt Generator

SEO Disclaimer: This tool generates standard robots.txt directives. Always verify the output before deploying to production servers. Misconfiguration may prevent search engines from crawling your site.

Easily create a custom robots.txt file to control how search engine crawlers access your website.

Hold Ctrl or Command to select multiple agents

What is robots.txt?

The robots.txt file tells web robots (search engine crawlers) which pages or files the bot can or cannot request from your site.

This standard helps manage crawler traffic and prevents unnecessary server load by specifying which areas of your site should not be processed or scanned.

Common Directives

  • User-agent: Specifies the robot this section applies to
  • Disallow: Prevents the crawler from accessing specific paths
  • Allow: Overrides Disallow for more specific subpaths
  • Crawl-delay: Specifies how many seconds the crawler should wait between requests
  • Sitemap: Points to your XML sitemap location
  • Host: Used in conjunction with sitemaps to specify the preferred domain

Example robots.txt

User-agent: Googlebot
User-agent: Bingbot
User-agent: Yahoo Slurp
Disallow: /admin/
Disallow: /temp/
Allow: /images/
Crawl-delay: 10

Sitemap: https://www.example.com/sitemap.xml 
Host: https://www.example.com 
            

Frequently Asked Questions

Where should I place robots.txt?

The robots.txt file must be placed in the root directory of your website (e.g., https://www.example.com/robots.txt ).

Do all bots respect robots.txt?

No. Most major search engines like Google, Bing, and Yahoo respect it, but malicious bots might ignore it completely.

How often do search engines check robots.txt?

Typically, search engines recheck robots.txt every 24 hours, though this can vary.

This tool provides educational content about robots.txt. Always test your configuration using Google Search Console or similar tools.