← All articles · SEO · about 3 min read
How to create a robots.txt file (with examples)
robots.txt is a small file at your site root that tells search bots what they may and may not crawl. The robots.txt Generator builds it for you — groups, rules, crawl-delay and sitemap — and lets you download it. Here's the basics.
Allow and disallow
Each block starts with 'User-agent' (a bot name, or '*' for all). Under it, 'Disallow' blocks a path; 'Allow' permits one. 'Disallow: /private/' keeps bots out of that folder; 'Disallow:' (empty) allows everything.
Crawl-delay and sitemap
Crawl-delay asks polite bots to wait between requests (handy on small servers). Always end with 'Sitemap: https://your-site.com/sitemap.xml' so bots find your index. Note: crawl-delay is ignored by Google (it uses Search Console settings) but respected by some others.
Generate, don't hand-type
Typos in robots.txt can block your whole site. Use the generator: add groups, write rules, set the sitemap, then download the file and place it at your domain root as '/robots.txt'.
Frequently asked questions
Does robots.txt enforce privacy?
No. It's a request, not a lock. Malicious bots ignore it, and blocked pages can still appear in indexes via links. For real hiding, use auth or noindex.
Where does robots.txt go?
At your domain root: https://your-site.com/robots.txt. A file in a subfolder only applies to that subfolder's path.
Is robots.txt required?
No, but it's useful to guide crawlers, block staging areas, and point bots to your sitemap. Without it, bots crawl what they find.