Robots.txt Generator
Build a ready-to-paste robots.txt, including rules for the AI crawlers you want to allow or block.
Tick a crawler to add a rule blocking it. Left unticked, it stays allowed.
User-agent: *
Disallow:
What the file you generate does
A robots.txt file tells crawlers which paths they may request. The generator writes a wildcard rule for ordinary crawlers, points at your sitemap, lists any paths you want disallowed, and adds a named group for each AI crawler you choose to allow or block. Those named groups matter: a crawler that matches its own group ignores the wildcard group entirely, so a rule you put only under the asterisk does not apply to it.
The AI crawlers worth naming
| User agent | What it does |
|---|---|
| GPTBot | OpenAI, model training |
| OAI-SearchBot | OpenAI, the ChatGPT search index |
| ChatGPT-User | Fetches a page live when someone asks ChatGPT about it |
| ClaudeBot | Anthropic |
| PerplexityBot | Perplexity's search index |
| Google-Extended | Gemini training. Does not affect Google Search or AI Overviews |
| CCBot | Common Crawl, the open dataset many models are pretrained on |
When this is worth running
When you are launching a site and want a correct file rather than one copied from a template you have not read. When you are deciding your position on AI crawlers and want that decision written down somewhere checkable. And when your existing file has grown by accretion, since most robots.txt problems come from a rule somebody added years ago that nobody has re-read since.
What robots.txt does not do
It is not access control and it is not a way to keep a page out of search. Disallow stops a compliant crawler from fetching a page, but the URL can still appear in results when other pages link to it, and Google cannot see a noindex tag on a page it is not allowed to fetch. To keep a page out of the index, allow the crawl and use a noindex robots meta tag instead. Nothing in the file is enforced either: a crawler that ignores it simply ignores it.
Questions people ask about robots.txt
Does blocking a page in robots.txt remove it from Google?
- No, and it can have the opposite effect. Google may keep showing the URL with no description, because it cannot fetch the page to read the noindex tag that would have removed it. Allow the crawl and use noindex instead.
Should I block AI crawlers?
- It depends on what you sell. Blocking them keeps your content out of model training and, for the search-focused ones, out of the answers those products cite. If being cited in AI answers is a channel you want, blocking the crawler closes it.
Where does the file have to live?
- At the root of the host, exactly at /robots.txt. It applies to that host and protocol only, so every subdomain needs its own file.
Other free tools
Why SEOryon
Free tools are the start. SEOryon does the whole job.
These tools give you a quick answer. SEOryon runs the full loop for you every week: it finds what you should write using real search data, then writes the content that ranks in Google and gets cited in AI answers.
Know what to write
Topics chosen from live search data, not guesswork, so your effort goes only where you can win.
Built to rank and get cited
Content written to rank in Google and to be cited in AI answers like ChatGPT, Perplexity, and Gemini.
You stay in control
Review and approve every piece before it goes live. Nothing publishes without your say.
Faster than doing it yourself, and a fraction of the cost of an agency, without handing over control of your brand.