Do AI bots fetch llms.txt?
In this 30-day sample, AI/AI-ish crawlers made 17,127 requests across applyr.co and seellm.com. They fetched robots.txt and sitemap paths 1,891 times. They made 0 requests to /llms.txt.
Window
May 1 to May 30, 2026
Domains
2
AI/AI-ish requests
17,127
/llms.txt AI hits
0
Short answer
The file was fetched by non-AI tools. We did not observe AI crawlers fetching it.
After excluding SeeLLM-AIVisibilityChecker/1.0, /llms.txt was fetched five times total across the two domains. All five returned 200.
The sources were BuiltWith, Dataprovider.com, and curl/8.7.1. Those are useful to separate from AI crawlers: they are bots or tools, but they are not evidence that an answer engine requested the file.
Fetch sources
The /llms.txt requests came from non-AI tooling.
| Source | Domain | Requests | AI crawler? |
|---|---|---|---|
| BuiltWith | seellm.com | 2 | No |
| BuiltWith | applyr.co | 1 | No |
| Dataprovider.com | applyr.co | 1 | No |
| curl/8.7.1 | seellm.com | 1 | No |
Strict AI
AI-classified requests
16,177
AI requests to robots.txt
1,191
AI requests to sitemap paths
573
AI requests to /llms.txt
0
Expanded AI-ish
AI/AI-ish requests
17,127
AI/AI-ish requests to robots.txt
1,317
AI/AI-ish requests to sitemap paths
574
AI/AI-ish requests to /llms.txt
0
Domain view
Both domains showed the same /llms.txt result.
| Domain | Strict / expanded | Robots | Sitemap | /llms.txt | Non-AI fetches |
|---|---|---|---|---|---|
| applyr.co | 14,431 / 15,095 | 870 / 958 | 365 / 365 | 0 / 0 | BuiltWith, Dataprovider.com |
| seellm.com | 1,746 / 2,032 | 321 / 359 | 208 / 209 | 0 / 0 | BuiltWith, curl/8.7.1 |
What this does not prove
This does not prove llms.txt is useless. It does not prove AI systems will never use it. It does not prove no AI crawler anywhere fetches it.
It only says that in this 30-day sample, on these two domains, AI and AI-ish crawlers fetched robots.txt and sitemap paths repeatedly, but made zero requests to /llms.txt.
- This does not prove llms.txt is useless.
- This does not prove AI systems will never use it.
- This does not prove no AI crawler anywhere fetches it.
- It only describes what appeared in these two domains' logs during this 30-day window.
Methodology
Window: May 1 to May 30, 2026. Domains: applyr.co and seellm.com. We excluded SeeLLM-AIVisibilityChecker/1.0 so our own scans did not count as external interest.
The strict AI count used bot_category IN ('ai_assistant', 'ai_training', 'ai_coding_agent', 'ai_referral'). The expanded AI-ish count added Applebot and DeepSeekBot rows that appeared outside the strict category but still represented AI-relevant crawler activity.
For /llms.txt, we separated AI crawlers, non-AI bots and web intelligence tools, browser or human traffic, and SeeLLM's own checker. The five remaining /llms.txt requests came from non-AI tools. None came from AI crawlers.
Practical takeaway
Do not treat file presence as crawler evidence.
Keep llms.txt if it helps documentation or future agent standards. For AI visibility, measure which bots actually request which files and pages.