Crawled, Cited, or Ignored? A Practical Framework for Measuring AI Visibility

AI visibility is not one metric. It spans access and readability checks, confidence-limited classified requests, sampled or platform answer evidence, detectable referrals, and instrumented actions.

Most teams skip straight to tactics.

Add schema. Rewrite the intro. Publish FAQs. Create comparison pages. Add an llms.txt file. Update metadata. Build more content.

Some of those actions can help. But none of them answer the first measurement question:

What evidence do you have for each page, and how was it collected?

If you cannot separate accessibility and readability checks from confidence-limited classified requests, sampled or platform retrieval and citation evidence, detectable referrals, and inferred states, you cannot interpret whether the measured evidence changed after a content update.

This framework is for measuring the state of a page before deciding what to fix.

AI visibility has layers

Treat AI visibility as a sequence, not a score.

The mistake is treating these layers as interchangeable.

A page can be accessible while no classified crawler request is observed in the stated window.

A page can be crawled but not cited.

A page can appear in a citation sample while producing no detectable referral in the same window. Because referrers can be stripped, that referral count is a lower bound.

A page can receive AI referral traffic from a system that fetched the source days earlier, or from an index rather than a live page request.

Each layer needs its own evidence.

Google is one layer, not the whole map

Google's guidance for AI Overviews and AI Mode is clear: the same SEO foundations still matter. Google says pages need to meet the normal technical requirements for Search, be indexed, and be eligible to show with a snippet. It also says there are no special AI markup requirements for those Google Search AI features.

That is useful guidance.

It also has a boundary: it is guidance for Google Search.

The broader AI web includes systems with different retrieval paths:

• Google AI Overviews and AI Mode, which are rooted in Google Search systems.
• Bing and Copilot experiences, where Microsoft now exposes AI citations, grounding queries, and page-level citation activity in Bing Webmaster Tools.
• ChatGPT search, where OpenAI distinguishes OAI-SearchBot for search from GPTBot for training and ChatGPT-User for user-triggered browsing.
• Claude, where Anthropic distinguishes ClaudeBot, Claude-User, and Claude-SearchBot.
• Perplexity, where PerplexityBot and Perplexity-User have different jobs.
• Browser agents, which may inspect screenshots, raw HTML, the DOM, and the accessibility tree.

That is why "AI visibility" cannot be reduced to one Google report, one crawler, or one optimization checklist.

The useful unit is the page

Site-wide averages hide the work.

For a SaaS, publisher, marketplace, or ecommerce site, a domain-level classified-request count is rarely enough.

The useful question is:

Which important pages had classified crawler requests in the stated window, and what separate sampled, platform, or detectable-referral evidence exists for each?

Start with pages where AI reuse would matter:

• homepage
• pricing
• product pages
• comparison pages
• category pages
• documentation entry points
• support pages
• high-intent editorial pages
• free tools and templates
• pages that changed recently

Then assign each page a job.

A pricing page should help a buyer understand plans, limits, and commitment level.

A comparison page should help someone choose between alternatives.

A documentation page should help an agent or user complete implementation.

A category page should define the problem, criteria, and tradeoffs.

If the page job is vague, the measurement will be vague too.

The page-level questions to ask

For each important URL, ask the questions in order.

1. Can AI systems access it?

Check the basics first:

• Does the preferred URL return a clean 200?
• Is the canonical URL correct?
• Is the page blocked by robots.txt?
• Is it blocked by noindex, X-Robots-Tag, or snippet controls?
• Is the page blocked by WAF, bot protection, geofencing, or login walls?
• Is the page linked from the site in a way crawlers can discover?
• Is it present in the sitemap if it should be?

Access is not success. It is the starting condition.

2. Which AI systems request it?

Look at server-side logs, CDN logs, or edge logs.

Do not rely only on browser analytics. Many crawler and fetcher requests never execute JavaScript analytics. They arrive as HTTP requests, receive the page, and leave no normal browser session behind.

Track at least:

• user agent
• URL
• timestamp
• status code
• referrer if present
• IP or ASN where available
• whether the bot identity was verified against published IP ranges

User-agent strings are useful, but they can be spoofed. Verification matters when you are making decisions from the data.

3. Can the system read the useful content?

A server fetch proves only the classified request, subject to confidence. It does not prove the content was parsed, retrieved for a task, cited, quoted, or easy to use.

Review the page from multiple machine-readable views:

• raw HTML
• rendered DOM
• visible text
• accessibility tree
• structured data where relevant
• important text inside images, widgets, tabs, modals, or scripts

For browser agents, web.dev recommends thinking beyond text extraction. Agents may use screenshots, raw HTML, and the accessibility tree. That means semantic buttons, labels, stable layouts, and clear interactive elements matter.

For search-grounded systems, text still matters. Google explicitly recommends making important content available in textual form for its AI features in Search.

4. Did the page get indexed or included in a retrieval surface?

Indexing is not the same as crawling.

A crawler can fetch a page without the page becoming useful in an answer system.

Use platform-specific tools where they exist:

• Google Search Console for Google indexing and Search performance.
• Bing Webmaster Tools for Bing crawl, index, and AI Performance data.
• URL inspection tools to confirm what the search system saw.
• Sitemaps and internal links to confirm discoverability.

For Bing and Copilot-style AI experiences, Bing's AI Performance dashboard is especially useful because it reports citations, cited pages, grounding queries, and visibility trends.

For other AI products, the evidence may be less complete. That is why log-level measurement and manual answer checks still matter.

5. Is the page cited, summarized, or referred to?

Crawler logs provide request evidence, subject to classification confidence. Documented citation or quoted-text samples and detectable AI referrals provide separate outcome evidence.

Look for:

• the URL appearing as a cited source in a documented answer sample
• the page's text matching a sampled answer under a stated rule and confidence
• detectable AI referral requests to the page
• changes in those sampled or detectable outcomes after a page update, without assuming causation from timing
• related pages receiving detectable AI referrals while this URL is absent from the specified citation sample
• query-to-page patterns in tools that expose them

Do not assume silence means failure. Some AI systems may use indexed information without creating a fresh fetch near the user session. Some answers influence buyers without sending a click. But repeated classified crawler requests combined with the URL's absence from a specified citation sample in the same window are a useful diagnosis trigger—not proof of universal non-citation or its cause. Treat quoted-text matches and detectable referrals as separate evidence.

Common patterns

Once you measure pages instead of domains, recurring patterns appear.

A manual measurement workflow

You can start without specialized tooling.

1. Pick 10 to 50 important URLs.
2. Confirm each page is accessible, canonical, indexable, and internally linked.
3. Fetch each page as raw HTML and confirm the main content is present.
4. Review server or CDN logs for known AI user agents.
5. Verify high-value bot traffic with published IP ranges where possible.
6. Group requests by page, bot, and week.
7. Compare AI referrals by landing page in analytics.
8. Check Google Search Console and Bing Webmaster Tools for page-level visibility.
9. Manually test a small set of buyer questions in AI search products.
10. Record each evidence state: no classified request observed, classified request observed, crawled but absent from the specified citation sample, cited in the sample, detectably referred, or changed.

The output should be an action list, not a dashboard screenshot.

Examples:

• Pricing is accessible, but no request classified as an AI crawler was observed in the stated window.
• Docs receive classified crawler requests weekly, but product pages have none in the same window; the pattern does not prove preference.
• The comparison page has classified crawler requests, is absent from the specified citation sample, and has no detectable AI referral in the stated window.
• The category page received detectable AI referrals after the latest rewrite; the timing alone does not establish causation.
• The setup guide appeared in the citation sample, but the related pricing page did not.

What to fix after measuring

Only fix the page state you can see.

If a page is not accessible, fix technical access.

If a page is accessible but has no classified crawler request in the stated window, investigate discovery: sitemap, internal links, canonicalization, navigation, and crawl permissions. The absence of an observed request does not prove no request occurred or establish the cause.

If a page is fetched but hard to parse, fix machine readability: visible text, semantic HTML, headings, labels, and stable layouts.

If crawler evidence exists but the URL is absent from the specified citation sample, investigate extractability alongside query fit, competing sources, and sampling conditions. Treat clearer definitions, answer-first sections, evidence, comparison criteria, and specific tradeoffs as hypotheses to verify with later citation or quoted-text samples.

If a page appears in the citation sample but has no detectable AI referral, review whether the cited answer gives users a reason to continue. Treat that as a hypothesis because stripped referrers make measured referrals a lower bound.

If an agent needs to act on the page, review the interface: buttons, forms, labels, error states, account requirements, and whether the next step is obvious.

The point is not to optimize every page for every AI system.

The point is to separate access checks, classified crawler requests, sampled citations and quoted text, detectable referrals, and inferred diagnosis states for important pages.

Sources worth using

These are useful starting points for building your own measurement model:

• Google's guide to optimizing for generative AI features on Search
• Google's AI features and your website guide
• Bing Webmaster Tools AI Performance dashboard
• OpenAI crawler documentation
• Anthropic crawler documentation
• Perplexity crawler documentation
• web.dev guide to building agent-friendly websites
• Cloudflare's guide to detecting AI crawlers

Where SeeLLM fits

You can do a basic version of this with logs, spreadsheets, Search Console, Bing Webmaster Tools, and manual checks.

That is often enough to document the observed requests, sampled outcomes, and inferred gap without claiming universal non-citation or a content cause.

SeeLLM is built to make the page-level workflow easier: choose the pages that matter, classify crawler requests subject to confidence, compare them with sampled citations and quoted text, report detectable AI referrals separately, and flag inferred gaps for diagnosis.

Start with the free AI Visibility Score to check whether an important page is technically readable. For the reuse gap, read What Is Crawled But Not Cited?. For the page-level operating workflow, read How to Monitor Important Pages for AI Reuse. For an empirical look at why splitting by platform matters — including a ~3× swing in Claude vs ChatGPT preference between two real domains — see what 30 days of AI bot traffic on two real domains actually looks like.