Robots.txt Generator
Generate robots.txt file
Frequently Asked Questions
What is a robots.txt file?
robots.txt tells search engine crawlers which pages to index and which to skip. It sits at your domain root (example.com/robots.txt). The generator creates properly formatted rules for common scenarios like blocking admin pages or specific bots.
How do I block search engines from indexing a page?
Add a Disallow rule: User-agent: * followed by Disallow: /private/. This blocks all crawlers from the /private/ directory. You can target specific bots (Googlebot, Bingbot) with separate User-agent sections.
Should I block CSS and JavaScript files in robots.txt?
No. Google recommends allowing access to CSS and JS files so it can render pages properly for indexing. Blocking these files can hurt your search rankings because Google cannot see your page as users do.
How do I add my sitemap to robots.txt?
Add a Sitemap directive at the end: Sitemap: https://example.com/sitemap.xml. This helps search engines discover your sitemap without relying on Search Console submission. Multiple sitemap URLs can be listed.
Does robots.txt guarantee pages will not be indexed?
No. robots.txt is a directive, not an enforcement mechanism. Well-behaved crawlers respect it, but malicious bots may ignore it. For guaranteed exclusion from search results, use the noindex meta tag or X-Robots-Tag HTTP header.