Robots.txt for AI crawlers: what to allow
By the Pickgap team · Updated Oct 15, 2026 · 3 min read
AI companies run separate crawlers for search and for training. To be a source in AI answers, allow the search crawlers, such as Googlebot, Bingbot, OAI-SearchBot and PerplexityBot. Blocking training crawlers like GPTBot or Google-Extended is a separate choice that doesn't remove you from search. Also check your CDN's bot settings, which can block crawlers your robots.txt allows.
Search crawlers vs. training crawlers
Most AI companies now separate their bots by purpose:
- Search crawlers fetch pages so an assistant can cite them in answers. Block these and your site can't be a source.
- Training crawlers collect content to train future models. Blocking them doesn't affect whether you're cited in search answers today.
- User-triggered fetchers visit a page when a person asks the assistant to read it.
For a local business that wants to be recommended, the decision is simple: allow search crawlers. Whether to allow training crawlers is a separate question of preference.
The main user agents
| Company | Search and answers | Training | Notes |
|---|---|---|---|
| Googlebot | Google-Extended | AI Overviews and AI Mode use Googlebot. Google-Extended covers Gemini model training and grounding, not Search. | |
| Microsoft | Bingbot | Bing's index powers Copilot and feeds other services. | |
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User fetches pages when a user asks. |
| Perplexity | PerplexityBot | Perplexity-User fetches pages when a user asks. | |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User fetches pages when a user asks. |
| Apple | Applebot | Applebot-Extended | Applebot-Extended controls training use. |
Crawler names change from time to time. Check each company's documentation before making big changes.
A template
This robots.txt allows everything, which is the right default for most local businesses:
User-agent: *
Allow: /
Sitemap: https://www.yourbusiness.example/sitemap.xmlIf you'd rather not contribute to model training but still want to be found and cited:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.yourbusiness.example/sitemap.xmlNever add a rule like "User-agent: OAI-SearchBot / Disallow: /" or "User-agent: PerplexityBot / Disallow: /" unless you want to be left out of those assistants' sources.
Check your CDN and firewall
Robots.txt is only a request. Your hosting may block bots before they ever read it:
- Some CDNs and security services offer a setting to block AI crawlers, sometimes turned on by default for new sites. Make sure it allows the search crawlers you want.
- Bot-fight or challenge modes can block crawlers that can't solve a challenge.
- Website builders may have their own "AI crawler" toggle.
After any change, test: fetch your homepage with a tool that shows the response for a given user agent, or check your server logs for visits from the crawlers above.
What robots.txt can't do
- It doesn't remove what an AI has already learned in training.
- It doesn't stop other sites from writing about you; AI can still cite them.
- It isn't a security tool. Anything private should be behind a login, not just disallowed.
What about llms.txt?
llms.txt is a proposed file that summarizes a site for AI systems. It's cheap to add, but none of the major AI search providers has said it uses llms.txt when deciding what to cite. See does llms.txt help SEO?
Questions
- If I block GPTBot, will ChatGPT stop recommending me?
- No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot. Blocking GPTBot keeps your content out of future training; it doesn't remove you from search answers.
- Do AI crawlers respect robots.txt?
- The major companies say their crawlers follow robots.txt. User-triggered fetchers, which act for a person who asked about a specific page, may be treated differently; check each company's documentation.