First-party research

Monthly reports on what AI bots actually do.

Every month we publish one finding from SeeLLM's first-party crawler dataset. Real bot fetches, real domains, methodology disclosed. No vendor spin, no scraped third-party stats.

Library

Every report we've shipped

In progress

Does llms.txt actually work? 30 days of crawler data

We measured how often the major AI bots request /llms.txt across SeeLLM's monitored domains. The answer settles a long-running r/SEO debate.

30 days of AI crawler logs across N sites

Planned

Each AI bot prefers a different kind of page

Pricing, comparison, docs, and product pages all get crawled - but not by the same bots. We split the data by bot to show what each LLM thinks your site is for.

Per-bot path-type distribution across SeeLLM corpus

How we report

Methodology, not marketing.

Every report includes the dataset window, the bot list, the IP/ASN cross-checks we used to dedupe spoofed user-agents, and the SQL or pseudocode for anyone who wants to run the same numbers on their own logs. We name the limits.

  • Dated observations- every chart names its date window. Stats that ship today won't still be true in 90 days; we say so.
  • Bot identification - user-agent matching plus IP/ASN reverse-DNS verification. Ambiguous fetches are dropped, not silently bucketed.
  • Sample size disclosed - number of sites, number of fetches, time window, on every report.
  • Caveats named - if our sample skews to mid-market SaaS, we say so. If a finding could be coincidence, we flag it.