robots.txt generator for AI crawlers

Why AI crawlers need their own robots.txt decisions

AI answer engines rely on their own crawlers, and each engine names a distinct user-agent in robots.txt. OpenAI alone runs three: GPTBot for training and answers, OAI-SearchBot for ChatGPT's web-search index, and ChatGPT-User for the live browsing that happens when a person asks a question. Anthropic splits ClaudeBot from Claude-Web, Perplexity runs PerplexityBot to build the index its answers draw from, and Google-Extended and Applebot-Extended separately control how Google's and Apple's AI products may use your content. A single old-fashioned "Disallow: /" under User-agent: * quietly refuses all of them - a common reason a brand vanishes from Perplexity or ChatGPT without anyone noticing a change.

The generator above gives you per-bot Allow/Block switches, three sensible presets, and a live preview of valid directives. If you are unsure where to start, "allow answers, block training" is the honest middle path: OAI-SearchBot, ChatGPT-User and PerplexityBot can still read you and cite you, while the training-centric crawlers hold off. Or allow everyone and be maximally available to every engine - for most businesses that want to appear in AI answers, the open file is the right default.

How the directives are assembled (the part people get wrong)

robots.txt is read in groups. A group starts at a User-agent: line and continues until the next agent line; consecutive agent lines share one rule-set, which is why the builder can merge your per-bot toggles into tidy blocks. Within a group, the longest matching path rule wins - "Disallow: /blog" outranks "Allow: /", and an explicit "Allow: /" on a strong group keeps your whole site readable while narrower Disallow entries carve out exceptions. A bot named by a group ignores the * group entirely: "User-agent: GPTBot" with a single Disallow line means Googlebot's permissions never apply to OpenAI. The generated file keeps each named bot in its own group so there is no accidental inheritance, puts your sitemap URL(s) at the end, and preserves your admin/cart/checkout exclusions under * so Googlebot and Bingbot keep obeying them.

Publish it, then prove it worked

Download the file and place it at your domain root so that visiting https://yourdomain.com/robots.txt shows exactly what you meant to publish. Then close the loop with the AI crawler check - it reads your live file and shows, bot by bot, who is actually allowed in. A rule you meant one way but wrote another is the classic mistake; the check exists because "I published it" is not evidence.

The file ages - review it like any crawl surface

New AI agents appear monthly, and each arrives with a user-agent line of its own: bytespider from ByteDance and the newest entrants keep the list above from being exhaustive. Every quarter or so, run the AI crawler check against your live domain and read the wayback of your own robots.txt decisions; if a rule you barely remember publishing for an engine now matters, patch it in this builder and deploy the regenerated file. The maintenance habit is the point - a robots.txt is one of the few SEO artifacts where a single line of fine text can opt a product in or out of a distribution channel overnight.

One honest limit: robots.txt is a request, not a wall. Well-behaved engines obey it, but it does not authenticate, prevent retrieval of pages already crawled, or stop an end-user from visiting your site through an agent. Treat it as a set of instructions you renew deliberately, and if a crawler misbehaves without respect for it, edge-level bot management is the enforcement layer - not this text file. For everything else, the trial scan tracks whether the engines now see and cite you.