AI crawlers
By the Pickgap team · Updated Oct 15, 2026 · 1 min read
AI crawlers are automated bots that AI companies send to read websites. Some collect content to train models (like GPTBot and ClaudeBot), some index pages so assistants can cite them in search answers (like OAI-SearchBot and PerplexityBot), and some fetch a page when a user asks. Websites control them with robots.txt and server or CDN settings.
Three kinds
| Kind | Purpose | Examples |
|---|---|---|
| Search crawlers | Index pages so they can be cited in answers | OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot |
| Training crawlers | Collect content for training models | GPTBot, ClaudeBot, and the Google-Extended and Applebot-Extended controls |
| User-triggered fetchers | Read a page when a user asks | ChatGPT-User, Perplexity-User, Claude-User |
How they're controlled
- robots.txt: each crawler checks for rules under its user agent name. Major companies say their crawlers follow these rules.
- CDN and firewall settings: can block bots before they reach your site, sometimes by default.
- Some controls are tokens, not separate bots: Google-Extended and Applebot-Extended are names you use in robots.txt to opt out of certain AI uses, while the normal crawler keeps visiting.
What a local business should do
Allow search crawlers and user-triggered fetchers, so AI can read and cite your site. Decide separately whether to allow training crawlers. Check your CDN settings. The details and a template are in robots.txt for AI crawlers.
A note on JavaScript
Several AI crawlers read a page's HTML without running JavaScript. Content that only appears after scripts run may be invisible to them. See does website speed matter for AI search?