Robots.txt Generator
Generate a robots.txt file for your website. Control search engine and AI bot access with an easy visual builder.
Quick Presets
Disallow:
Allow:
/
Generated robots.txt
User-agent: * Allow: /
How to use: Save this as robots.txt in your website's root directory (e.g., https://example.com/robots.txt).
Frequently Asked Questions
What is robots.txt?+
robots.txt is a text file placed in your website's root directory that tells search engine crawlers which pages they can and cannot access. It follows the Robots Exclusion Protocol standard.
Does robots.txt block pages from appearing in Google?+
No. robots.txt prevents crawling, not indexing. Google can still index a URL if it finds links to it, even without crawling the content. To prevent indexing, use a noindex meta tag instead.
Should I block AI bots with robots.txt?+
It depends. If you don't want your content used for AI training, you can block bots like GPTBot, ClaudeBot, and PerplexityBot. However, blocking them also prevents your content from appearing in AI-powered search answers.