🪓 robots.txt Generator
Build a valid robots.txt with user-agent rules, allow/disallow paths, crawl-delay and a sitemap. Copy or download, then drop it at your site root. Local only, no upload.
❓ Frequently asked questions
Does this host my robots.txt?
No. It generates the text for you to upload to the root of your own site. A robots.txt only works at https://yoursite.com/robots.txt.
What is the difference between Allow and Disallow?
Disallow tells crawlers not to fetch matching paths; Allow (used by Google and others) permits a sub-path inside a disallowed area. Specificity and order matter.
Is my input uploaded?
No. The file is built locally in your browser.
📚 How to use
- Pick a preset or add agent groups, then type the allow/disallow paths.
- Add your sitemap URL(s) if you have one.
- Copy or download the result and upload it to your site root.
About this tool
robots.txt is the first file most search engines read. A single wrong line — blocking / by accident — can drop your whole site from search. This generator keeps the syntax valid so you only worry about what to allow.
Group rules by user-agent (use * for all crawlers), list the paths to allow or block, set a crawl-delay if you want to slow bots down, and finish with your sitemap so engines find every page.
Nothing is uploaded — the file is assembled in your browser as you type.