Just as Googlebot crawls the web for Google Search, AI companies run their own crawlers to gather what their assistants know and cite: OpenAI’s GPTBot, Anthropic’s ClaudeBot, PerplexityBot, and Google’s Google-Extended token, among others. They read your pages and decide whether your content is worth surfacing in an answer.
Here is where businesses quietly shoot themselves in the foot. Some website setups, security plugins, or "block AI" settings turn these crawlers away by default in robots.txt. That can feel protective, but for a local business trying to get recommended by AI, blocking the crawler is like unplugging your phone and wondering why nobody calls. If the bot cannot read you, the assistant cannot name you.
The judgment call is real: content owners with something to protect (a paywalled archive, proprietary data) may choose to block. But a small business that wants to be found should almost always let the reputable AI crawlers in, and then make sure what they read is clean, accurate, and structured to be quoted.
A single line in robots.txt, "User-agent: GPTBot / Disallow: /", quietly removes your business from the pool ChatGPT can draw on. Many owners have that line and have no idea it is there.
Go deeper: How to Show Up in ChatGPT and Perplexity Results (2026).
This is part of our AI search (GEO) work.