Robots.txt Generator
Build a robots.txt file, including rules to block AI training crawlers.
How to use this tool
- Select your file, drag it onto the box above.
- Processing starts automatically — there is nothing to configure.
- Check the result panel, which shows the original size, the final size and the page count.
- Click Download to save it.
robots.txt is advisory. Well-behaved crawlers obey it; others ignore it entirely. It is not a security measure, and disallowing a path can even advertise its existence.
The specification
| Presets | Allow all, block all, or custom |
|---|---|
| Rules | Disallow and Allow paths, crawl-delay |
| AI crawlers | Optional block for GPTBot, CCBot, ClaudeBot, Google-Extended and others |
| Sitemap | Reference added at the end |
| Location | Must sit at your domain root |
How it works
This tool runs entirely in your browser. There is no server involved, which means nothing you type or upload is transmitted, stored, or logged — and there are no usage limits, because there is nothing for us to ration.
You can confirm it yourself: open your browser's developer tools, select the Network tab, and use the tool. No request is made.
Who uses this
Anyone launching a site who needs a correct robots.txt, or wanting to opt out of AI training crawlers.
Frequently asked questions
Where does robots.txt go?
At the root of your domain — example.com/robots.txt. It has no effect anywhere else, and subdomains need their own.
Does disallowing a page remove it from Google?
No, and this catches people out. Disallow stops crawling, not indexing — a blocked page can still appear from external links. To remove a page, allow crawling and use a noindex meta tag.
Should I block AI crawlers?
Your call. Blocking may reduce your presence in AI-generated answers, which for some sites is meaningful traffic. Blocking Google-Extended does not affect normal Search ranking.
Are my files uploaded to a server?
No. Everything runs inside your browser on your own device. Your file is never transmitted and never stored by us. You can confirm this by opening your browser’s Network tab while you use the tool — you will see no upload request.
Is there a limit on how many files I can process?
No. There is no daily cap, no account, and no watermark. Because the work happens on your device rather than our servers, there is nothing for us to ration.
Is this tool really free?
Yes — no account, no trial, no watermark, and no cap on usage. It runs on your own device rather than our servers, so there is no cost for us to pass on.
Does it work on mobile?
Yes. The interface adapts to phone and tablet screens, and everything runs in the browser, so there is no app to install.
Related tools
From other categories
Your files stay on your device. This tool runs entirely in your browser. No file is uploaded, transmitted, or stored. You can verify this by opening your browser's Network tab while using the tool.
Disclaimer. Specifications published by exam boards, governments and platforms change without notice. We verify and date every specification on this site, but you should always confirm against the official source before submitting an application. See our full disclaimer.