Are AI crawlers allowed?
Blocking AI bots in robots.txt is the most common reason a site never appears in AI answers. Check your site’s permissions in one click.
Up to eleven engines · 23 markets · refreshed daily
Before any of this matters, the crawler has to get in. This reads your robots rules, your rendering and your access log to see whether it did.
First report ready in 4 minutes
Can AI crawlers read your site? A blocked site cannot benefit from any content work.
✓ Start audit| Status | Check | Finding | Open task |
|---|---|---|---|
| blocking | llms.txt | No llms.txt. Not required, but a cheap way to describe the site’s structure to AI crawlers. | Open task |
| fix this | Sitemap | The sitemap has not been updated for 47 days. Freshness matters especially to Perplexity. | Open task |
| fix this | Response speed (TTFB) | Time to first byte 840 ms. Slow responses burn crawl budget. | Open task |
| passing | robots.txt — AI crawlers | Every AI crawler can read the site. | — |
| passing | Content without JavaScript | The content is present in the raw HTML. | — |
| passing | Structured data (JSON-LD) | Schemas found: Organization, Product, FAQPage. | — |
| passing | Bot protection (CDN/WAF) | Bot protection is not blocking AI crawlers. | — |
The scan runs when you start it, with “Start audit” or from a workflow; fixes are re-verified on the next scan you start. Tasks you open go to you or your developer as instructions.
A surprising number of sites block the crawlers behind AI answers without ever deciding to — a robots rule copied years ago, a bot filter at the CDN, or content that only exists after JavaScript runs. The check names which AI crawler is allowed, which is refused, and which pages came back empty when fetched the way a model fetches them. Where you have a log drain or Cloudflare connected, it also shows which crawler actually arrived, on which URL, and when.
Rixe collects a large amount of evidence about how AI assistants describe brands. Here is what exists, and how to get at it.
Two sources, both your own. Search Console and Google Analytics 4 supply clicks and sessions; your access log or CDN supplies the crawler hits themselves. Nothing is modelled or estimated from a third-party panel.
The AI crawlers that actually reached the page, matched by user agent — so you see which crawler hit which URL and when, not an estimate of how often you were probably crawled.
The most expensive line in a robots.txt is the one nobody remembers writing. A rule added years ago against scrapers, a CDN bot filter switched on during an incident, a firewall default — any of them can make a site invisible to answer engines while search traffic carries on as if nothing happened.
Answer crawlers also come for different reasons, and the difference matters when you decide what to allow. One builds the index an assistant answers from. One is a live fetch with a person waiting on the other end. One collects training data for a model that ships next year. Blocking the second breaks an answer in progress; blocking the third costs you nothing today.
Reachability is only half of it. A page the crawler fetched but could not read — because the text arrives after JavaScript runs — is present and useless, and it will not appear in the citation list no matter how good it is. The check reports both, because they have different owners and different fixes.
Blocking AI bots in robots.txt is the most common reason a site never appears in AI answers. Check your site’s permissions in one click.
What did the crawler come for?
In your sitemap but no bot has read it. Unread pages cannot appear in answers.
The crawler asked for these and got an error. Failures on live user fetches come first — an answer is breaking as it happens.
On the left, how many pages the engine read; on the right, how many people arrived from it. An engine that reads everything and sends nobody is a different problem from one that never comes.
Visits to your site from AI assistants. This is where visibility converts to traffic.
This measures movement together, not cause. Nothing in this data can separate it from a good month for the whole category.
| Assistant | AI share | Session | Conversion |
|---|---|---|---|
| ChatGPT | 49% | 1,468 | 41 |
| Perplexity | 24.9% | 745 | 39 |
| Gemini | 15.5% | 465 | 35 |
| Microsoft Copilot | 10.6% | 319 | 39 |
Each of these is a step the product runs, not a line on a slide — the name is the one it carries in the workflow builder.
Blocking AI bots in robots.txt is the most common reason a site never appears in AI answers. Check your site’s permissions in one click.
Queues a technical audit for the selected pages.
No source is attached yet. If you are on Cloudflare, connect it in one click.
Adds the selected pages to the action list as refreshes.
Can AI crawlers read your site? A blocked site cannot benefit from any content work.
Visits to your site from AI assistants. This is where visibility converts to traffic.
Measured gaps become briefs, briefs become drafts; nothing is published until you approve it.
Six screens carry the whole job: read the answer, score it, find the sources behind it, and turn the gap into work someone can pick up.
A full crawl of your own site: on-page issues, broken links, duplicate titles and Core Web Vitals. No external cost.
Open page →Everything AEO, GEO and SEO found, as one ranked list — split by who does the work.
Open page →The sites and pages engines rely on when answering.
Open page →A full crawl of your own site: on-page issues, broken links, duplicate titles and Core Web Vitals. No external cost.
Crawl the siteFrom Google's PageSpeed Insights. Field data where enough Chrome users visited, lab data otherwise.
| URL | Performance | LCP | CLS | INP | Core Web Vitals |
|---|---|---|---|---|---|
| / | 91 | 1.9s | 0.03 | 140ms | |
| /pricing | 84 | 2.4s | 0.06 | 180ms | |
| /integrations/gmail | 88 | 2.1s | 0.02 | 160ms | |
| /blog/crm-for-real-estate-teams | 72 | 3.1s | 0.12 | 230ms |
/blog/crm-migration-checklist · /docs/api-v1 · /integrations/outlook …
/features/forecasting-beta · /partners/old
/blog/pipeline-stages · /blog/sales-forecasting · /templates …
/ · /integrations · /customers …
/blog/best-crm-for-small-business · /compare/hubspot · /blog/gmail-crm …
/templates/sales · /templates/support · /templates/hr …
/pricing · /security · /about
/changelog/2026-08 · /legal/dpa
/lp/webinar-2025 · /lp/black-friday · /beta
Everything AEO, GEO and SEO found, as one ranked list — split by who does the work.
HubSpot beats you on 104 questions. "X vs Y" and "X alternative" pages are what assistants quote directly on the comparison questions buyers ask most.
We write and publish the page · Generate a comparison brief for this rival from the Competitors page; the rival’s side is written only from collected, real sources.
Pipedrive beats you on 15 questions. "X vs Y" and "X alternative" pages are what assistants quote directly on the comparison questions buyers ask most.
We write and publish the page · Generate a comparison brief for this rival from the Competitors page; the rival’s side is written only from collected, real sources.
Zoho CRM beats you on 13 questions. "X vs Y" and "X alternative" pages are what assistants quote directly on the comparison questions buyers ask most.
We write and publish the page · Generate a comparison brief for this rival from the Competitors page; the rival’s side is written only from collected, real sources.
monday CRM beats you on 4 questions. "X vs Y" and "X alternative" pages are what assistants quote directly on the comparison questions buyers ask most.
We write and publish the page · Generate a comparison brief for this rival from the Competitors page; the rival’s side is written only from collected, real sources.
Salesforce beats you on 58 questions. "X vs Y" and "X alternative" pages are what assistants quote directly on the comparison questions buyers ask most.
We write and publish the page · Generate a comparison brief for this rival from the Competitors page; the rival’s side is written only from collected, real sources.
On gemini-grounding, Salesforce appears 100% of the time and you appear 0%.
We write and publish the page · Write a page that answers this question directly: a clear answer in the first paragraph, a table, numbers, and the date it was updated.
On perplexity-sonar, Salesforce appears 100% of the time and you appear 0%.
We write and publish the page · Write a page that answers this question directly: a clear answer in the first paragraph, a table, numbers, and the date it was updated.
The sites and pages engines rely on when answering.
| Source | Type | Used | Citations | Trend | |
|---|---|---|---|---|---|
| g2.com | You | 100% | 10,210 | Stable | Create action |
| reddit.com | You | 100% | 7,293 | Stable | Create action |
| flowcrm.io | You | 94% | 5,108 | Stable | Create action |
| capterra.com | You | 88% | 4,471 | Stable | Create action |
| techradar.com | You | 71% | 2,960 | Stable | Create action |
Brands and agencies track their AI visibility with Rixe
AI agents now open sites to book or buy for a person. The audit checks whether robots.txt lets ChatGPT-User, Claude-User, Perplexity-User and Google-Agent in, whether your forms can be filled in by an agent, and whether buttons and links can be found. WebMCP, an emerging proposal, is reported for information only.
In the app: AI access & pages › Audit
See which AI bots read your pages and what they got back, from Cloudflare, the WordPress plugin, a log drain or an uploaded server log. Visits are grouped by purpose (live user fetch, search index, training crawl), checked against the address ranges each company publishes, and listed per page, with a badge when AI answers cited it. Only real visits are shown, never estimates.
In the app: Sources & AI Traffic › Crawlers
No. The robots and rendering checks need only your domain. Seeing which crawler actually arrived needs a log drain or the Cloudflare connection — one read-only token, no code on the site.
The AI crawlers that fetch pages for answers, matched by user agent, so the report names which one reached which URL and when rather than estimating how often you were probably crawled.
Your robots.txt and, if you connect one, your access log — looking for the named AI crawlers and the status code each one got.
Any host you can point at, as long as it answers publicly. A staging site behind basic auth reads as blocked, which is correct rather than a bug.
Yes, robots.txt alone tells you whether they are allowed. The log tells you whether they actually arrived, which is the more common surprise.
Managed rules. A CDN's bot-fighting preset or a security vendor's default can block AI agents without anyone in your team choosing that.
Not always. A paywalled publisher may decide otherwise. For anyone selling a product, being unreadable means being unrecommendable.
It removes the hard stop. The engine still has to crawl and then choose your page, which takes days to weeks rather than hours.
No. Speed is a search concern; this is about access and readability, which is what decides whether the page exists for a model at all.
After every CDN, WAF or robots change, and once a quarter regardless. Access breaks silently, and nothing else on the site announces it.
Three numbers, one screen. Enough to know which way the week went.
What's the best project management tool for small teams?
What are the alternatives to Asana?
The CRMs that come up most often for small teams are HubSpot, Pipedrive and FlowCRM. FlowCRM is the one people pick when the team already lives in Gmail.
Can you recommend a CRM with multilingual support?
The CRMs that come up most often for small teams are HubSpot, Pipedrive and FlowCRM. FlowCRM is the one people pick when the team already lives in Gmail.
Which reporting tool do agencies use?
What's the best project management tool for small teams?
What are the alternatives to Asana?
The CRMs that come up most often for small teams are HubSpot, Pipedrive and FlowCRM. FlowCRM is the one people pick when the team already lives in Gmail.
Can you recommend a CRM with multilingual support?
The CRMs that come up most often for small teams are HubSpot, Pipedrive and FlowCRM. FlowCRM is the one people pick when the team already lives in Gmail.
Which reporting tool do agencies use?
Which tool has the most generous free plan?
Jira or Linear — which one is easier?
Which software is the fastest to set up?
Best analytics tool for e-commerce?
Which tool has the most generous free plan?
Jira or Linear — which one is easier?
Which software is the fastest to set up?
Best analytics tool for e-commerce?
Which tool is GDPR compliant?
What should a new company start with?
What is the best CRM for a small team?
Which tool is GDPR compliant?
What should a new company start with?
What is the best CRM for a small team?
“Being well-ranked on Google is no longer the same as being recommended by ChatGPT. It's good to have tools that try to measure this difference.”See how
AI engines on the menu, each measured on its own surface.
markets, each asked in its own language.
measurement, with a confidence range on every score.
Search Console, Analytics, Shopify, Slack, Notion, Linear — and anything else your team and your agents run on.
Rixe asks the AI engines your buyers use the questions they actually ask, every day, and shows which names come back and why.
Stay as involved as you like: the plan says what to fix, and on managed plans Rixe carries it out.