# llms.txt in 2026: who actually reads it (mostly coding agents) > Ahrefs found 97% of llms.txt files got zero requests in a month, and Google Search ignores them. Who does read llms.txt, when it pays off, how to write one. Source: https://seofix.ai/blog/does-llms-txt-work Published: 2026-10-26 Updated: 2026-10-09 Author: SEOFix Team Category: AI search Tags: AI agents, Guides Length: 1,805 words, 8 min read For AI search visibility, llms.txt doesn't work, or at least nobody has shown that it does. No major AI search crawler documents reading it, Google Search says it doesn't use such files, and in Ahrefs' study of 137,210 domains, 97% of the sites that published an llms.txt saw no requests for it at all in May 2026 ([Ahrefs](https://ahrefs.com/blog/llmstxt-study/)). The requests that did arrive came mostly from SEO tools and agent tooling, and Anthropic's coding agent Claude Code fetched it more than any AI search or training bot except GPTBot. So llms.txt is worth publishing if developers point coding agents at your docs, and not worth much effort otherwise. ## What llms.txt is, and what it isn't llms.txt is a proposal by Jeremy Howard, first published on 3 September 2024, for a Markdown file at `/llms.txt` that gives language models a short, curated map of a site ([llmstxt.org](https://llmstxt.org/)). The format, from the spec: 1. An H1 with the name of the project or site. The spec calls this ["the only required section"](https://llmstxt.org/). 2. A blockquote with a short summary. 3. Optional paragraphs or lists with more detail. 4. Sections under H2 headings, each a list of links with a short description. 5. By convention, a section called `Optional` for ["links an agent can skip when a shorter context is needed"](https://llmstxt.org/). The spec also suggests serving a Markdown version of each page at the same URL with `.md` appended. It's a reading aid, not a directive. Unlike robots.txt, it doesn't allow or block anything, and no crawler is obliged to read it. Anything you put in it is your own description of your own site, which is why John Mueller of Google argued in June 2026 that LLM systems, ["by design, can't trust what is here as a way of differentiating between different websites"](https://www.searchenginejournal.com/googles-mueller-says-llms-txt-cant-help-llms-differentiate-sites/). He did see one narrow use: ["If someone is already on your website, maybe some kind of automated system is helpful"](https://www.searchenginejournal.com/googles-mueller-says-llms-txt-cant-help-llms-differentiate-sites/). ## Does llms.txt work? The evidence ### Ahrefs: 137,210 domains, one month of requests Ahrefs published the largest study we know of on 15 June 2026, based on May 2026 data from Ahrefs Web Analytics ([Ahrefs](https://ahrefs.com/blog/llmstxt-study/)): - 28% of the 137,210 domains published a valid llms.txt. - Of those ~38,000 files, ["97% saw no requests for it whatsoever in May"](https://ahrefs.com/blog/llmstxt-study/). - 96% of the requests that did happen came from bots. - The largest group of those bot requests was SEO audit tools (21.7%), then unidentified bots (14.9%), general web crawlers (13.1%) and tech profiling tools (11.6%). AI agents and infrastructure made up 10.5%, training crawlers 5.3%, AI assistants 2.5% and retrieval bots such as OAI-SearchBot and PerplexityBot 1.1%. - In Ahrefs' words: ["Aside from statespace-indexer and GPTBot, Claude-Code (Anthropic's coding agent), outfetched every AI retrieval bot, every AI assistant, and every AI training crawler."](https://ahrefs.com/blog/llmstxt-study/) Ahrefs' verdict was that ["the cons outweigh the pros right now"](https://ahrefs.com/blog/llmstxt-study/) for anyone chasing AI search visibility. A disclosure that matters for reading those numbers: SEO audit tools fetch llms.txt because they check whether it exists. SEOFix does too. Every audit requests `/llms.txt` for its `LLMS_TXT_MISSING` check. So the biggest category in Ahrefs' data says more about audit tools than about AI. ### What the vendors say - **Google Search:** its AI optimization guide (updated 10 July 2026) says ["You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) Mueller had already said in 2025: ["FWIW no AI system currently uses llms.txt."](https://www.seroundtable.com/google-ai-llms-txt-39607.html) - **Chrome:** a different Google team went the other way. Lighthouse's Agentic Browsing category has an llms.txt audit, last updated 5 May 2026, which says that without the file ["agents may spend more time crawling the site to understand its high-level structure"](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt). It only fails on a server error. A missing file (404) is marked not applicable. Search Engine Journal covered the [two positions side by side](https://www.searchenginejournal.com/googles-llms-txt-guidance-depends-on-which-product-you-ask/). - **OpenAI, Anthropic, Perplexity:** their crawler documentation ([OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518), [Perplexity](https://docs.perplexity.ai/guides/bots)) says nothing about GPTBot, ClaudeBot, OAI-SearchBot or PerplexityBot reading llms.txt. We checked on 8 October 2026. ### Who publishes one Developer documentation is where llms.txt caught on, because coding agents read docs. On 8 October 2026 we requested `/llms.txt` from a few well-known docs sites: | Site | `/llms.txt` | `/llms-full.txt` | | --- | --- | --- | | developers.openai.com | 200, 6 KB | not checked | | platform.claude.com (docs.claude.com redirects there) | 200, 83 KB | 200, 42 MB | | nextjs.org/docs | 200, 51 KB | not checked | | docs.stripe.com | 200, 92 KB | not checked | | developers.cloudflare.com | 200, 17 KB | 200, 67 MB | | developers.google.com | 404 | not checked | OpenAI's, Perplexity's and Next.js's doc pages each carry a pointer to their index. Next.js's says ["For an index of all Next.js documentation, see /docs/llms.txt"](https://nextjs.org/docs/app/api-reference/functions/generate-metadata). An agent reading one page learns where the index is. That fits Mueller's "already on your website" case and Ahrefs' coding-agent data much better than the idea of AI search engines checking `/llms.txt` on every domain. ## What about llms-full.txt? `llms-full.txt` is a convention that grew up next to the spec: the full text of every linked page concatenated into one file. The llmstxt.org page doesn't define it. It's useful when an agent should load a whole small doc set at once, and it stops being useful as it grows. Anthropic's docs serve 42 MB and Cloudflare's 67 MB. An agent can't sensibly read a file that size in one go. It has to search it or fetch pieces, and the link list in `llms.txt` already lets it fetch pieces. Our rule of thumb: publish `llms-full.txt` only if the whole thing is small enough that reading it end to end is a reasonable thing for an agent to do. Ours is 437 KB. ## How seofix.ai serves llms.txt seofix.ai is a product people connect to coding agents over MCP and a REST API, so we're in the group that might benefit. What we publish, generated from the same data as the site (`web/lib/marketing/llms.ts`): - **`/llms.txt`** (13 KB, 60 links): an H1, a one-paragraph blockquote, a paragraph with the free audit limit and the price range of each plan (built from the pricing config, so it can't drift from the pricing page), then a `Docs` section, one `Help: ` section per help-center category, a `Blog: ` section per blog category, an `API` section with the REST and MCP endpoints, and an `Optional` section with the legal pages. - **Every link points at Markdown.** Help, legal and blog pages have `.md` twins (`/help/blocked-pages.md`), and the normal URL returns Markdown when a client sends `Accept: text/markdown`. - **`/llms-full.txt`** (437 KB): the summary, then every help article and legal page in full. - Both are served as `text/plain; charset=utf-8`. `/llms.txt` is rebuilt when a blog post is published, and at least hourly. Here is the shape, trimmed: ```markdown # SEOFix > SEOFix is an agent-first technical SEO audit service. It crawls a website politely, groups every issue by page template, writes the exact fix for each, and hands the report to AI coding agents (Claude Code, Codex, Cursor) over a REST API and an MCP server. The first audit is free, up to 500 pages, with no card needed. ... ## Docs - [Help center](https://seofix.ai/help): every article below as one file at https://seofix.ai/llms-full.txt; JSON index at https://seofix.ai/help.json. Append .md to any help or legal URL (or send Accept: text/markdown) for plain markdown. - [MCP server setup](https://seofix.ai/mcp): connect Claude Code, Codex or Cursor with an API key; lists all 17 MCP tools ## Help: Sites & firewalls - [Let SEOFix through your firewall](https://seofix.ai/help/firewall-allowlisting.md): Allowlist SEOFixBot in Cloudflare or any other WAF ... ## Optional - [Terms of Service](https://seofix.ai/legal/terms.md): The agreement between you and Inhype Live Limited for using SEOFix: ... ``` **What we'll measure.** We don't have usage numbers yet; the files haven't been live long enough. Over the next 30 days we'll count requests to `/llms.txt`, `/llms-full.txt` and the `.md` pages by user agent and publish the breakdown here: how many come from coding agents, AI crawlers, SEO tools and everything else, and whether any of it turns into signups. Until then we won't claim it does anything for us. ## When it's worth 20 minutes **Worth it:** - Developer docs, API references, SDKs, CLIs. Your users ask Claude Code, Cursor or Codex to "read the docs for X", and an index of Markdown pages saves the agent from scraping HTML navigation. - Products used through agents (MCP servers, APIs), where the agent needs setup steps, auth and limits. - Large help centers where a curated list of the 30 pages that matter beats the agent guessing from a sitemap of 3,000. **Not worth much:** - Marketing sites, blogs, ecommerce and local businesses hoping to appear in ChatGPT, Perplexity or AI Overviews. For those, what matters is whether AI crawlers can fetch your pages at all and whether the content is in the server HTML. - Anything where you'd write it once and never update it. A stale llms.txt with dead links is worse than none for an agent that trusts it. ## A template for a developer-facing llms.txt ```markdown # Acme API > Acme is a REST API for sending transactional SMS. Base URL https://api.acme.dev/v1, Bearer-token auth, JSON in and out. Rate limit: 50 requests per second per key. Use the Markdown links below; every docs page also exists at the same URL with .md appended. ## Getting started - [Quickstart](https://docs.acme.dev/quickstart.md): create a key, send a first message with curl, read the delivery webhook - [Authentication](https://docs.acme.dev/auth.md): key types, scopes, rotation ## API reference - [Send a message](https://docs.acme.dev/api/messages-create.md): POST /v1/messages, every parameter and error code - [Webhooks](https://docs.acme.dev/api/webhooks.md): event types, signature verification, retries - [Errors](https://docs.acme.dev/api/errors.md): every error code with the fix ## SDKs - [Node.js](https://docs.acme.dev/sdk/node.md): install, client setup, examples - [Python](https://docs.acme.dev/sdk/python.md): install, client setup, examples ## Optional - [Changelog](https://docs.acme.dev/changelog.md): API changes by date - [Pricing](https://acme.dev/pricing.md): per-message prices by country ``` What makes a good one: - **Put the facts an agent gets wrong in the blockquote.** Base URL, auth scheme, rate limits, the current version. These are what an agent otherwise guesses. - **Describe each link.** "every parameter and error code" tells the agent which page to open. A bare list of titles doesn't. - **Link to Markdown, not HTML.** If you don't have `.md` versions of your pages, an llms.txt pointing at HTML still helps, but much less. - **Generate it from your content.** A build step or a route that reads your docs index won't go stale the way a hand-written file does. - **Serve it as text.** Send `text/plain` or `text/markdown` with a `200`. A single-page app whose catch-all route answers `/llms.txt` with the app's HTML shell is a common way to "have" an llms.txt that's really a 200 HTML page. In Next.js, a route handler at `app/llms.txt/route.ts` does it: ```typescript // app/llms.txt/route.ts import { buildLlmsTxt } from '@/lib/llms' export const dynamic = 'force-static' export const revalidate = 3600 export async function GET() { return new Response(await buildLlmsTxt(), { headers: { 'Content-Type': 'text/plain; charset=utf-8' }, }) } ``` Then check what you actually serve: ```bash curl -sI https://example.com/llms.txt | grep -i -E '^(HTTP|content-type)' curl -s https://example.com/llms.txt | head -5 ``` You want `200`, a text content type, and a first line starting with `# `. SEOFix's `LLMS_TXT_MISSING` check does the same test on every audit: it reports the file as missing when `/llms.txt` doesn't answer `200` with a non-empty, non-HTML body, so an SPA fallback page counts as missing. It's a **notice**, the lowest severity we have, because the evidence above doesn't justify more. The check and its fix are in [AI visibility and rendering checks](https://seofix.ai/help/checks-ai-and-rendering). If you want to see how an agent uses our own docs, the [API quickstart](https://seofix.ai/help/api-quickstart) and [MCP server](https://seofix.ai/help/mcp-server) pages are the ones our llms.txt points coding agents to first. ## FAQ ### Does llms.txt help with ChatGPT or Perplexity rankings? There's no evidence it does. Neither OpenAI nor Perplexity documents reading it for search, and Ahrefs measured retrieval bots such as OAI-SearchBot and PerplexityBot at 1.1% of bot requests to llms.txt files ([Ahrefs](https://ahrefs.com/blog/llmstxt-study/)). What decides whether those products can cite you is whether their crawlers are allowed in robots.txt and your CDN, and whether your content is in the server HTML. ### Does Google use llms.txt? Google Search says it doesn't ([Google](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)). Chrome's Lighthouse checks for one in its Agentic Browsing category, which is about browser agents, not Search ([Chrome](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt)). ### Can llms.txt block AI training? No. It has no allow or disallow semantics. Use robots.txt groups for `GPTBot`, `ClaudeBot`, `Google-Extended` and similar, as described in [what to block and allow in 2026](https://seofix.ai/blog/block-gptbot-robots-txt). ### Should llms.txt list every page? No. The spec describes it as a curated overview. List the pages an agent needs to do a job, and use the `Optional` section for the rest. --- Want to check llms.txt and AI-crawler access in one go? [Run a free audit](https://seofix.ai/): both checks run on every audit, including the 50-page preview.