Key takeaways

  • llms.txt is a proposal from Jeremy Howard and Answer.AI, revised to version 2 in August 2026, with no ratification from a standards body.
  • Google Search ignores the file, and an Ahrefs study of 137,210 domains found that 97 percent of published files drew no requests in May 2026.
  • Coding agents and docs platforms do use it, which makes it worth publishing for developer documentation and not worth treating as a ranking lever.

The llms.txt standard is not a standard the way HTTP is. It is a proposal by Jeremy Howard of Answer.AI, first published on September 3, 2024 and revised to version 2 in August 2026, asking sites to put a Markdown index at /llms.txt. Almost every argument about it grows from that gap: a convention with wide tooling behind it, no ratification from a standards body, and no commitment from a major search engine to read it.

What the proposal actually specifies

The format fits in a paragraph. A file at /llms.txt, or at a subpath like /docs/llms.txt, opens with an H1 name, which is the only required part. A blockquote summary follows, then optional context, then H2 sections of links with notes after a colon. An ## Optional section holds pages an agent may skip. Version 2 adds link relations: rel="alternate" type="text/markdown" for a Markdown twin of a page, and rel="describedby" pointing back at the file.

The document calls itself a proposal and keeps its discussion in a public GitHub repository. That is a normal way to start a convention. It is not the same as a specification with a conformance suite and a committee behind it.

The case for it

Adoption is real, and it sits where agents already work. OpenAI, Anthropic and Google publish llms.txt files for their developer docs. Mintlify, GitBook, Yoast, AIOSEO and Wix generate one. Chrome's Lighthouse audits for the file as part of its agentic browsing checks, though a missing file counts as not applicable rather than a failure. When a coding agent needs your API reference, a short curated index beats a page wrapped in navigation and cookie banners.

The case against, in numbers

Google is the strongest data point against treating the file as a visibility lever. Its AI optimization guide says Search needs no new machine-readable files, and that maintaining an llms.txt will neither help nor hurt. In June 2025, John Mueller said server logs showed no AI system using the file. In May 2026, Ahrefs examined logs for 137,210 domains: about 28 percent had a valid file, a figure Ahrefs calls an upper bound because its customers skew technical, and 97 percent of those files drew no requests that month. GPTBot made the most fetches and Claude Code came second.

Claim What the evidence supports
Agents read llms.txt Some do, mostly coding agents, mostly for docs
Google rewards it No. Search ignores it
It replaces robots.txt No. Access rules stay in robots.txt
It replaces a sitemap No. It is a reading list, not an inventory
Every site needs one Sites with docs, an API or product facts assistants get wrong

Both camps overreach. "No AI system uses it" described AI search crawlers in 2025 logs and says little about the coding agent that opened your docs this morning. "Agents need it" ignores that nearly all published files went unread.

What to do while the argument runs

Publish one if you have documentation worth indexing, keep it accurate, and spend the rest of the effort on what Google does read: crawlable pages, server-rendered content and specific facts. A validator run at seogeoaeo.ai costs 10 credits and catches the syntax and link faults that make an agent's visit worthless. That is a smaller promise than the debate usually makes, and a more useful one.

One line of housekeeping for the people who will link this page in a quote tweet: the file is a proposal, not a law, and the absence of a standards body cuts both ways. It means no one can fail you for a missing file, and also that an agent team can change its mind about reading yours without asking anyone.

FAQ

Is llms.txt an official standard?

No. It is a proposal from Answer.AI, published at llmstxt.org and revised to version 2, with no ratification from a standards body.

Will Google penalize a missing llms.txt?

No. Google says its search ignores the file, so a missing one neither helps nor hurts. Lighthouse treats a missing file as not applicable.

Who actually reads it?

Mostly coding agents and assistants working with developer documentation. No major AI search engine has said it uses the file to choose citations.

Frequently asked questions

Is llms.txt an official standard?

No. It is a proposal from Answer.AI, published at llmstxt.org and revised to version 2, with no ratification from a standards body.

Will Google penalize a missing llms.txt?

No. Google says its search ignores the file, so a missing one neither helps nor hurts. Lighthouse treats a missing file as not applicable.

Who actually reads it?

Mostly coding agents and assistants working with developer documentation. No major AI search engine has said it uses the file to choose citations.

Sources

  1. The /llms.txt file, v2llmstxt.org
  2. Ahrefs study of llms.txt requestsAhrefs
  3. Google's guide to optimizing for generative AI in SearchGoogle
  4. AI Agent Readiness Checkerseogeoaeo.ai
  5. Lighthouse llms.txt auditChrome for Developers

Add ClaudeGPT Si as a preferred source on Google