First-party research

Monthly reports on what AI bots actually do.

Every month we publish one finding from SeeLLM's first-party crawler dataset. Real bot fetches, real domains, methodology disclosed. No vendor spin, no scraped third-party stats.

Run the free page scan Read the latest report

Latest report

May 2026

Do AI bots fetch llms.txt? A small server-log check

We checked 30 days across applyr.co and seellm.com. AI/AI-ish crawlers made 17,127 requests and fetched robots/sitemaps 1,891 times, but made 0 requests to /llms.txt.

Authority asset: 30 days - 2 domains - 17,127 AI/AI-ish requests

Library

Every report we've shipped

One report per month. Each anchored on a specific question, a dataset slice, and a methodology you can replicate.

April 2026Published

What AI bots actually do on your site

A 7-day snapshot of seven AI crawlers across three live domains. Each bot has a different personality, and none of it shows up in GA4.

7 bots × 3 sites × 7 days behavioral data

June 2026Planned

Each AI bot prefers a different kind of page

Pricing, comparison, docs, and product pages all get crawled - but not by the same bots. We split the data by bot to show what each LLM thinks your site is for.

Per-bot path-type distribution across SeeLLM corpus

How we report

Methodology, not marketing.

Every report includes the dataset window, the bot list, the IP/ASN cross-checks we used to dedupe spoofed user-agents, and the SQL or pseudocode for anyone who wants to run the same numbers on their own logs. We name the limits.

Dated observations- every chart names its date window. Stats that ship today won't still be true in 90 days; we say so.
Bot identification - user-agent matching plus IP/ASN reverse-DNS verification. Ambiguous fetches are dropped, not silently bucketed.
Sample size disclosed - number of sites, number of fetches, time window, on every report.
Caveats named - if our sample skews to mid-market SaaS, we say so. If a finding could be coincidence, we flag it.