SEO
Robots.txt Generator
Build a production-ready robots.txt with allow/disallow rules, crawl-delay, and sitemap references — validated as you type.
User-agent groups
Generated robots.txt
User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
FAQ
Where do I upload robots.txt?+
It must live at the root of your domain — e.g. https://example.com/robots.txt. Subdirectory placement is ignored by crawlers.
Does Disallow guarantee no indexing?+
No. It only blocks crawling. To prevent indexing, use a noindex meta tag or an X-Robots-Tag header on the page itself.
Should I include a sitemap?+
Yes — declaring your sitemap URL helps every crawler discover it, even those that don't check /sitemap.xml by default.