AI robots.txt and llms.txt Generator
Decide which AI assistants may read your website, then copy a clean robots.txt. Build an llms.txt that points AI tools to your most important pages.
Direct Answer
robots.txt tells each crawler what it may read. AI search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) decide whether you can be cited in AI answers; training bots (GPTBot, Google-Extended, ClaudeBot, CCBot) decide whether your content trains AI models. You can allow one group and block the other.
Your choices
Your file robots.txt
Downloads are disabled in this preview. Copy the text into a file.
Frequently asked questions
What is the difference between GPTBot and OAI-SearchBot?
+
Both belong to OpenAI. OAI-SearchBot crawls pages so ChatGPT search can cite and link to them. GPTBot collects content that may be used to train models. You can allow the first and block the second.
Does blocking Google-Extended remove me from Google?
+
No. Google-Extended only controls whether your content is used for Google's AI models. Googlebot, which ranks you in search and AI Overviews, is separate.
What is llms.txt?
+
A plain text file at yoursite.com/llms.txt that lists your key pages with short notes, so AI tools can find your most useful content quickly. It is new and optional, and cheap to add.
Where do I put these files?
+
Both go in the root of your domain: yoursite.com/robots.txt and yoursite.com/llms.txt.
Other free tools
Want an expert to fix it for you?
Send your website address. We review it for free and tell you what to fix first.