llms.txt: Created for language models — ignored by agents

2026-06-22 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

A September 2024 proposal by answer.ai, led by Jeremy Howard, introduced "llms.txt" as a text-based instruction file for language models. Intended to provide concise summaries and references, thereby saving context window space, recent analysis from Limy based on 515 million bot events reveals only 408 inquiries to "llms.txt" files. Despite its initial purpose, generalist agents like ChatGPT, Claude, and Gemini currently do not utilize "llms.txt" for training data, general search, RAG, or direct URL lookups. The primary exception is agentic code tools such as Claude Code, Copilot, Codex, and Cursor, which increasingly use it for navigating complex developer documentation. The lack of broader adoption is attributed to the overhead and increased costs for agent providers, alongside concerns about the file's potential for inaccuracy or manipulation.

Key takeaway

For web developers and content strategists considering "llms.txt" for general websites, you should currently avoid implementing it. Generalist agents like ChatGPT and Claude largely ignore these files, making the effort a waste of resources. Instead, focus your time on creating tidy content structures, using plain language, ensuring universal design, and employing clear semantic HTML. This approach significantly improves how language models interpret and present your content to users.

Key insights

"llms.txt", designed to optimize language model context, is largely ignored by generalist agents but adopted by agentic code tools.

Principles

• LLM providers prioritize cost efficiency and content authenticity over parsing supplementary files.
• Effective web content for LLMs relies on fundamental web standards and clear human language.

In practice

• Implement "llms.txt" for developer documentation to aid agentic code tools.
• Prioritize semantic HTML and clear navigation for general web content.

Topics

• llms.txt
• AI Agents
• Language Models
• Web Crawling
• Semantic HTML
• Developer Documentation

Best for: AI Architect, AI Product Manager, AI Engineer, Software Engineer, Director of AI/ML

Related on AIssential

• Large Language Models Do Not Always Need Readable Language — LLMs can recover semantics from compact, non-human-readable textual representations, decoupling readability from model-side understanding.
• Web search — Ollama's new web search API enhances LLMs by providing real-time web data to improve accuracy and reduce hallucinations.
• Does AGENTS.md Actually Help Coding Agents? — Context files for coding agents are beneficial only when they provide unique, non-redundant information.
• Building my own LLM-Wiki Research Team — Treat notes like code and compile them into a persistent, compounding markdown wiki using an LLM Agent to avoid rediscovering insights.
• I don't understand AI. How does it work? — LLMs predict text by mapping words to semantic vectors and learning statistical relationships from vast training data.
• Alternatives for the EDIT tool of LLM agents — Token-efficient LLM agent editing can be achieved via line-specific checksum tags, improving reliability and speed.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.