Latest dispatch
When Cloudflare Blocks AI Crawlers
Your analytics can look healthy while ChatGPT, Claude, and Perplexity cannot fetch your important pages.
AI visibilityAI crawlersCloudflareanalyticscontent operations
Field notes
Research, diagnostics, and opinionated playbooks around important pages, AI-mediated discovery, citations, and crawled-but-not-cited failures.
Latest dispatch
Your analytics can look healthy while ChatGPT, Claude, and Perplexity cannot fetch your important pages.
Archive
Practical writing for operators trying to understand what changed, where it changed, and what to fix next.
A practical measurement guide for separating AI bot visits, citation evidence, and referral sessions so content and SEO teams can see which AI tools actually send traffic.
SEO benchmarking tracks Google performance. AI visibility benchmarking documents sampled answer outcomes separately from classified crawler requests and detectable referrals.
We checked 30 days of server logs from two monitored domains. AI crawlers fetched robots.txt and sitemap.xml, but not llms.txt.
Four patterns from 30 days of AI bot traffic on a consumer content site and a B2B SaaS site — patterns that flat AI-traffic dashboards hide.
Crawled but not cited means crawler evidence exists, but the URL was absent from a specified citation sample.
A framework for separating access and readability checks, classified requests, sampled or platform answer evidence, detectable referrals, and inferences.
In a May 2026 test, SeeLLM scored 100/A on bot access and 67/100 on agent readiness because our H1 started 8,388 HTML characters into the page.
On one content site, AI bot fetches held steady while measured AI referrals fell 80× over 90 days. The decoupling — not the dramatic number — is the durable insight.
SEO rankings measure search discoverability. AI visibility combines classified crawler requests with sampled answer and detectable-referral evidence.
A practical page-level workflow for comparing classified crawler requests with sampled citations, quoted text, and detectable AI referrals.
AI visibility is becoming operational. Crawled but not cited compares classified crawler evidence with a specified citation sample.
Install the SeeLLM Cloudflare Worker and collect server-side evidence about requests classified as AI crawler traffic.
Google Analytics is useful for human sessions, but it cannot show which pages AI systems fetch, revisit, cite, skip, or leave crawled but not cited.
SeeLLM separates classified crawler requests, sampled citations, detectable referrals, and inferred page-level evidence states.