Why is 'allow or block' the wrong question for AI crawlers?
The useful question is what each crawler does with your content. Some AI crawlers collect material that may be used to train models. Others fetch pages so an assistant can search, read and cite them when a user asks a question. Treating these as one category forces a trade-off that does not need to exist.
For a B2B team, the cost of over-blocking is usually higher than the cost of allowing search access. If your product pages, comparison pages and documentation are invisible to answer engines, buyers who ask assistants for vendor shortlists will never see your brand. The goal is to decide, crawler by crawler, which purpose you accept.
A workable framework has three layers: training access, search and citation access, and user-initiated retrieval. Write one decision per layer, then express it in robots.txt. Revisit the decisions when vendors update their documentation, because crawler names and behaviour can change.
What are GPTBot, OAI-SearchBot and ChatGPT-User, and which should you allow?
GPTBot is OpenAI's crawler associated with training data collection. OAI-SearchBot is associated with ChatGPT search, so it indexes content that can appear in search answers. ChatGPT-User is used when a person asks ChatGPT to open a page during a conversation. Each has a different purpose and, according to OpenAI's crawler documentation, each is controlled differently.
A common decision for B2B brands is to disallow GPTBot, allow OAI-SearchBot, and treat ChatGPT-User as a user-initiated request. Check OpenAI's current documentation before you finalise rules, because the details of how each agent handles robots.txt are the kind of thing vendors revise.
| Crawler | Main purpose | Typical decision for a B2B site | What blocking does |
|---|---|---|---|
| GPTBot | Training data collection | Disallow if you opt out of training | Stops training collection; does not remove search visibility on its own |
| OAI-SearchBot | Indexing for ChatGPT search | Allow if you want citation | Blocking can keep your pages out of ChatGPT search results |
| ChatGPT-User | Fetching a page at a user's request | Usually a product decision, not a training one | May not be governed by robots.txt in the same way as crawlers |
Does blocking Google-Extended affect Google Search or AI Overviews?
Blocking Google-Extended does not remove your pages from regular Google Search, and according to Google's documentation it does not affect inclusion in AI Overviews. Google-Extended is a robots.txt token that controls whether content Google has crawled may be used for training Gemini models and for certain grounding uses. It is not a separate crawler that fetches your pages.
The risk is in confusing it with Googlebot. If you disallow Googlebot, your pages drop out of Search and AI features that rely on Search. Disallowing Google-Extended only signals that your content should not be used for Gemini training and related uses.
Before changing anything, confirm your Googlebot rules explicitly. A broad disallow for all user agents, copied from an old template, is the most common reason a site disappears from Google when the team only meant to opt out of AI training.
How do you write robots.txt rules that allow citation but opt out of training?
Write a separate User-agent group for each AI crawler you want to control, and keep Googlebot and Bingbot groups for normal search. Rules are matched per group, so a Disallow under GPTBot does not affect OAI-SearchBot. Keep the file at the root of each hostname, because robots.txt rules do not carry across subdomains.
The rules below are a starting pattern for a team that opts out of training but wants citation. Adjust them to your own position on training, and do not copy them without reviewing which paths you actually want to protect.
- User-agent: GPTBot / Disallow: /
- User-agent: Google-Extended / Disallow: /
- User-agent: OAI-SearchBot / Allow: /
- User-agent: PerplexityBot / Allow: /
- User-agent: Googlebot / Allow: /
- User-agent: * / Disallow: /account/ and /staging/
- Audit your current robots.txt and note any broad rules under the asterisk user agent, since these also apply to AI crawlers.
- Add a group for GPTBot with Disallow: / if you opt out of OpenAI training.
- Add a group for Google-Extended with Disallow: / if you opt out of Gemini training. Keep Googlebot unrestricted.
- Add a group for OAI-SearchBot and PerplexityBot with Allow: / so their search access is explicit.
- Protect private paths such as account areas, staging folders and internal search results with Disallow rules that apply to all agents.
- Test the file in your search console tools and in a robots.txt testing tool, then deploy and re-check after each site release.
Which AI crawlers respect robots.txt, and which may not?
Major AI operators that publish crawler documentation generally state that their declared crawlers honour robots.txt. Those statements are the baseline, but they are promises rather than guarantees, and each vendor's wording should be checked in its own documentation.
Robots.txt is advisory. It works because well-behaved crawlers choose to read it. Some user-initiated fetches, and some requests made through third-party tools, may not follow the same rules. Third-party reports have also alleged that certain answer engines fetched content using user agents that did not match their declared names. Treat any user agent string as a claim until you have verified it.
If you need enforcement, robots.txt alone is not enough. Use authentication for private content, rate limits or a web application firewall for abusive traffic, and be honest internally that a published rule is a signal, not a lock.
How do you verify that a PerplexityBot or GPTBot request is genuine?
Verification means checking that the request comes from an IP address that belongs to the claimed operator. A user agent string is easy to copy, so it proves nothing by itself. Major operators that run dedicated crawlers have published IP range lists, and Google documents how to verify Googlebot with a reverse DNS lookup followed by a forward lookup.
The check is simple to run from a shell or a log tool. Compare the IP addresses in your logs against the operator's published list, and treat any request that claims a name but comes from outside those ranges as unverified. Keep a record of the date you pulled each list, because ranges change.
- Extract requests whose user agent contains GPTBot, OAI-SearchBot, PerplexityBot, Perplexity-User or Googlebot from your server logs.
- Collect the unique source IP addresses for each claimed name.
- For each operator that publishes ranges, check whether the IP falls inside its listed ranges.
- For Googlebot, run a reverse DNS lookup on the IP and confirm the hostname belongs to Google, then run a forward lookup to confirm it resolves back to the same IP.
- Label each source as verified, unverified or no published list, and only trust crawl counts from verified sources.
- Repeat the check monthly, since new IP addresses and new user agents appear over time.
How do you confirm in server logs that AI crawlers reach your key B2B pages?
Server logs show what crawlers requested, when, and what status code your server returned. For a B2B site, filter those requests down to the pages that drive pipeline, such as product, pricing, integration, security and comparison pages, and check whether AI crawlers reach them. A crawler that never requests a key page cannot use it for citation, whatever your robots.txt says.
Look at three things: which AI user agents appear, which URL paths they request, and whether those requests return a 200 status. Frequent 403, 404 or 5xx responses to verified AI crawlers point to access problems, such as a firewall rule or a broken template, rather than a robots.txt decision.
- Export raw access logs for at least a representative period and confirm the time zone.
- Filter for AI user agents and separate verified from unverified requests using the method above.
- Count requests per URL path, grouping by page type such as product, article, documentation and legal.
- Check the status codes returned to each verified crawler and fix any 4xx or 5xx pattern.
- Compare the paths requested with the pages you want cited, and link the gaps to your internal linking and sitemap.
- Record the results so you can compare them after each change to robots.txt or site structure.
Where Rixe fits in
Manual log review is useful for a first audit, but it does not show how answer engines describe your brand for the questions buyers ask. An AI visibility platform such as Rixe measures how assistants mention and cite a brand for a set of prompts, which lets you check whether crawler access is translating into visibility.