Developer · Free forever
Robots.txt generator
Build a valid robots.txt file in seconds. Allow, disallow, sitemap, and crawl-delay supported.
Frequently asked questions
Where do I place robots.txt?
Upload the file to the root of your domain, so it is reachable at https://yourdomain.com/robots.txt. Search engines look there automatically.
Does Disallow guarantee a page stays out of Google?
No. It only asks compliant crawlers not to fetch the page. To exclude pages from the index reliably, use a noindex meta tag or HTTP header.
Should I always include a sitemap line?
Yes, if you have an XML sitemap. It helps crawlers find your URLs faster, especially on new domains.