Do AI Bots Fetch llms.txt? A Small Server-Log Check

llms.txt is a useful idea: give AI systems a simple, machine-readable file that explains the important parts of a site.

The open question is more practical:

Are AI crawlers actually fetching it today?

We checked 30 days of server-side logs from two domains monitored by SeeLLM:

• applyr.co
• seellm.com

This is a small sample. It is not a verdict on the future of the llms.txt proposal. But it is useful field evidence for one narrow question: in these logs, did detectable AI crawlers request the file?

Short answer

No AI crawler requested /llms.txt on either domain during the 30-day window.

After excluding SeeLLM's own checker, /llms.txt was fetched five times total:

All five returned 200.

So the file was reachable. It was just fetched by non-AI web intelligence tools, not by the AI crawlers we observed.

What AI crawlers did fetch

Across the same two domains, we saw substantial AI crawler activity.

Using the stricter AI classifier:

Using an expanded "AI-ish" cut that also includes Applebot and DeepSeekBot:

That distinction matters. The bots were not ignoring discovery files in general. They requested robots.txt and sitemap paths 1,891 times in the expanded cut.

They just did not request /llms.txt.

Domain-level view

applyr.co, May 1 to May 30, 2026:

After excluding SeeLLM's checker, applyr.co had two /llms.txt fetches. They came from BuiltWith and Dataprovider.com.

seellm.com, May 1 to May 30, 2026:

After excluding SeeLLM's checker, seellm.com had three /llms.txt fetches. They came from BuiltWith and curl.

What this does not prove

This does not prove llms.txt is useless.

It does not prove AI systems will never use it.

It does not prove no AI crawler anywhere fetches it.

It only says this:

In this 30-day sample, on these two domains, AI and AI-ish crawlers fetched robots.txt and sitemap paths repeatedly, but made zero requests to /llms.txt.

That is a narrower claim, and it is the claim the logs support.

Why this matters

A readiness checklist can tell you whether a file exists.

Server logs can tell you whether anyone used it.

Those are different questions.

If you publish llms.txt, it may still be worth doing. It can be useful documentation. It may help future agents. It may become more important if adoption changes.

But today, at least in this sample, llms.txt presence would have been a misleading success metric. The external non-AI fetches returned 200, but the detectable AI crawlers did not request the file.

For measurement, the safer question is not:

"Do we have llms.txt?"

It is:

"Which bots actually requested which files and pages?"

Methodology

Window: May 1 to May 30, 2026.

Domains: applyr.co and seellm.com.

We excluded requests from SeeLLM-AIVisibilityChecker/1.0 so our own scans did not count as external interest.

The strict AI count used SeeLLM's AI bot categories:

bot_category IN ('ai_assistant', 'ai_training', 'ai_coding_agent', 'ai_referral')

The expanded AI-ish count added Applebot and DeepSeekBot rows that appeared outside the strict category but still represented AI-relevant crawler activity.

For /llms.txt, we separated:

• AI crawlers
• non-AI bots and web intelligence tools
• browser or human traffic
• SeeLLM's own checker, which was excluded

The five remaining /llms.txt requests came from non-AI tools. None came from AI crawlers.

Practical takeaway

Do not remove llms.txt just because of this small sample.

But do not treat llms.txt as evidence that AI crawlers are reading your site, either.

If the goal is AI visibility, measure behavior:

• Did AI crawlers request the page?
• Did they request the supporting discovery files?
• Which bots came back?
• Which important pages did they ignore?
• Did crawl activity turn into citation or referral evidence?

That is the layer a static checklist cannot show.

For a broader measurement model, read Crawled, Cited, or Ignored? A Practical Framework for Measuring AI Visibility.