Should a local business block AI crawlers?
By the Pickgap team · Updated Oct 15, 2026 · 1 min read
Generally, no, not the search crawlers. A local business wants AI assistants to find, read and recommend it, and blocking search crawlers like OAI-SearchBot or PerplexityBot removes your site as a source. Blocking training-only crawlers like GPTBot is a reasonable personal choice and doesn't affect search answers. Check your CDN too: its bot settings may be blocking crawlers you never meant to block.
Why publishers block, and why that logic rarely fits you
News sites and publishers block AI crawlers because their content is the product, and AI answers can replace visits. A local business is different: your product is a service in the real world. An AI that reads your site and tells someone "they're open Saturday and take your insurance" is doing your marketing.
The sensible default
| Crawler type | Examples | Recommendation |
|---|---|---|
| Search engines | Googlebot, Bingbot | Allow |
| AI search | OAI-SearchBot, PerplexityBot, Claude-SearchBot | Allow |
| User-triggered fetchers | ChatGPT-User, Perplexity-User | Allow |
| Training | GPTBot, ClaudeBot, Google-Extended | Your choice |
Blocking training crawlers keeps your text out of future model training. It doesn't stop AI from citing you in search answers, and it doesn't remove what models already learned.
The hidden blocker: your CDN
Many sites block AI crawlers without anyone deciding to. CDNs and security services offer bot-blocking settings, sometimes on by default for new sites, and website builders may have their own toggle. These act before robots.txt is ever read. If AI assistants seem unable to read your site, check:
- Your CDN's bot and AI crawler settings.
- Any "bot fight" or challenge mode.
- Your website builder's SEO or crawler settings.
How to check
Read your robots.txt (yoursite.com/robots.txt), check the settings above, and look in your server logs for visits from the crawlers in the table. The full list and a template are in robots.txt for AI crawlers.