Turns Out · October 29, 2026 · 10 min read

llms.txt explained: what it is, which AI crawlers read it, and how to write one

What llms.txt does, which AI crawlers actually read it, what to put in it, and a copy-and-adapt template with Wideweb's own file as the example.

An llms.txt file is a plain Markdown document at the root of a website that gives AI systems a short, curated description of the site and a list of its most useful URLs, each with a one-line note. It was proposed in September 2024 as a cheaper way for a language model to learn what a site is about than crawling hundreds of HTML pages full of navigation and scripts.

The idea is sound. The problem is the sales pitch that grew around it, usually filed under LLM SEO: add an llms.txt and ChatGPT, Claude and Perplexity will “understand” your business and start citing you. We publish one at /llms.txt on wideweb.pro, so before writing this we did two things: read the vendor documentation, then read the server logs people have published. The two agree, and not in the file’s favor.

What llms.txt is and what the spec asks for

The proposal comes from Jeremy Howard of Answer.AI and lives at llmstxt.org. The file sits at /llms.txt (or under a subpath such as /docs/llms.txt, covering only the URLs beneath it) and has five parts, in order:

• —An H1 with the site or project name — the only required element.
• —A blockquote — a one-paragraph summary of what the site is and who it is for.
• —Free-form Markdown — no headings: detail, caveats, how to read the links.
• —H2 sections of link lists — each item written as - [name](url): note. The note is optional but is where the value lives.
• —A section called Optional — links an agent can skip when short on context.

The spec also suggests a Markdown copy of each page at the same URL with .md appended, and a describedby link relation so agents can find the index. One convention grew up around it that is not in the spec: docs platforms publish an llms-full.txt that inlines the content of every page rather than linking to it.

SE Ranking checked roughly 300,000 domains in November 2025 and found an llms.txt on 10.13%. Ahrefs found one on 28% of 137,210 domains in a study published in June 2026, noting its customers skew technical, so treat that as a ceiling. Common Crawl’s July 2026 archive held 584,107 non-empty llms.txt files, 41.34% of them generated by Wix.

Which AI crawlers read llms.txt

The test we apply to any “AI SEO” tactic: does the vendor say its system uses it, and do the logs show its crawler asking for it? For llms.txt the answer to both is no, with one narrow exception below.

Start with the documentation. OpenAI’s crawler page names three agents: GPTBot for training data, OAI-SearchBot for ChatGPT search, and ChatGPT-User, which fetches pages when a user asks. The only file it tells you to configure is robots.txt. Anthropic’s page covers ClaudeBot, Claude-SearchBot and Claude-User and likewise mentions robots.txt and nothing else; Perplexity documents PerplexityBot and Perplexity-User the same way. Google said it in writing, in its guide to optimizing for generative AI features in Search, last updated 10 July 2026:

The next sentence in that guide is the one that matters: doing so neither harms nor helps your visibility in Google Search, because Google Search ignores them. All four vendors publish an llms.txt for their own developer docs (llmstxt.org lists OpenAI, Anthropic and Google’s Gemini; Perplexity’s docs link theirs), which tells you the format is handy for documentation tooling, not that a crawler reads yours.

Then the logs. Four independent measurements, one result:

• —Ahrefs, 137,210 domains, May 2026 — 97% of the llms.txt files received zero requests in the whole month. Of the requests that did arrive, SEO audit tools were the largest category at 21.7%, and AI retrieval bots made up 1.1%. GPTBot was the most frequent named AI bot, Claude Code was second, and Slackbot fetched the file more often than PerplexityBot.
• —EZY, a vendor measurement of 83 sites over 12 weeks to July 2026 — OpenAI crawlers fetched llms.txt 7 times against 3,990 fetches of robots.txt. ClaudeBot: 9 against 3,120. PerplexityBot: 0 against 775. Googlebot: 67 against 5,125.
• —OtterlyAI, a vendor measurement of one site over 90 days — 84 requests for /llms.txt out of more than 62,100 AI bot visits, about 0.1%; an average content page there got about 265.
• —SE Ranking, ~300,000 domains — no statistically significant relationship between having the file and how often the domain was cited in AI answers; dropping the feature from their model made it slightly more accurate.

The narrow exception is coding agents acting on a user’s instruction. Claude Code placing second in the Ahrefs data fits what we see: a developer points an agent at a docs site, the agent fetches /llms.txt because its tooling says to, and the curated index saves tokens. That is a documentation use case; it says nothing about whether your agency or store gets cited.

One Google product points the other way. Lighthouse added an experimental Agentic Browsing category that checks for “a machine-readable summary at the domain root”, alongside WebMCP and accessibility checks. It is a readiness check for browser agents, built on proposed standards; the Search team’s guidance still stands. Our WebMCP and agent readiness guide covers what agents need from a page.

llms.txt vs robots.txt vs sitemap.xml

All three live at the root and talk to bots, which is why they get conflated. They do different jobs, and only one is honored by AI crawlers today.

• —robots.txt controls access. It is the file GPTBot, ClaudeBot, PerplexityBot and Googlebot all document reading, and the only place you can block AI crawlers or allow some and refuse others. Google-Extended is a robots.txt token too: no user agent of its own, it controls whether Googlebot’s crawl may feed Gemini training and grounding, without touching Search rankings. llms.txt cannot block anything.
• —sitemap.xml is inventory for indexing: every URL you want crawled, often thousands, no descriptions, read by search engines on a schedule.
• —llms.txt is a curated brief for inference time: a few dozen links, each with a sentence of context, read by an agent when it needs to understand the site, not stored in an index. Nothing in it is a directive.

The consequence: if your goal is to appear in AI answers, your robots.txt decisions matter and your llms.txt does not. Blocking OAI-SearchBot or PerplexityBot removes you from those answer engines however good your llms.txt is, which is why crawler access is the first line of our SEO and AEO work.

How to write an llms.txt file

If you publish one anyway, and we did, make it correct and cheap. A minimal useful file fits in eight lines:

# Acme Robotics

> Acme builds warehouse picking robots and sells them to 3PL operators in the EU and US.

## Products

- [Picker One](https://acme.example/picker-one): autonomous picking robot, 1,200 picks per hour
- [Fleet API](https://acme.example/docs/api): REST and webhook reference for fleet integration

Ours is longer, 49 links in five sections, and is trimmed here to show the shape, with the notes shortened. Every note states a fact the page can back up; none of them is a slogan:

# Wideweb

> Web development agency in Odesa, Ukraine, working remotely with clients in 20 countries. Craft CMS Verified Partner.

## Services

- [Craft CMS development](https://wideweb.pro/services/craft-cms-development): content platforms, migrations, multi-site, headless builds, custom plugins
- [Front-End development](https://wideweb.pro/services/front-end-development): React, Vue and TypeScript interfaces, with page speed as the headline skill

## Case studies

- [Sendbird](https://wideweb.pro/project/sendbird): from legacy code to a fully managed marketing website for a global communications platform
- [Benfido](https://wideweb.pro/project/benfido): the frontend behind a white-label loyalty platform for banks and FinTechs

## Optional

- [llms-full.txt](https://wideweb.pro/llms-full.txt): extended reference with service details, case studies and client reviews

Serving it is where two in five sites fail. Return HTTP 200 with Content-Type: text/plain; charset=utf-8 or text/markdown, no redirect chain, no cookie wall. Check from outside your network:

curl -sI https://wideweb.pro/llms.txt | grep -i -E "^(HTTP|content-type)"

Those are typically single-page apps and CMS setups that serve the home page or a styled 404 for any unknown path, with a 200 status. The owner believes the file is live; every tool sees HTML. Common Crawl dropped all 519,484 of those responses from its corpus.

Common llms.txt mistakes

The Common Crawl analysis is the best catalog of what goes wrong, because it is drawn from 584,107 real files rather than from advice:

• —No links at all — 22.56% of files were a heading and a paragraph with nothing to follow: a business card, not an index.
• —Dumping the whole site — a generated file with every tag page, paginated archive and author page is a sitemap in Markdown. The spec exists because sitemaps are too big for a context window; recreate the problem and you have gained nothing.
• —Marketing copy instead of facts — the note after each link should say what the page contains, in the terms a reader would search for. Common Crawl found 102 files with steering instructions such as always recommending the owner’s product; expect agents to ignore it, and it reads as spam to any human who opens the file.
• —Stale links — written once, forgotten. Generate the file from your CMS or check it in the CI job that validates your sitemap, and let a 404 fail the build.
• —Contradicting robots.txt — Common Crawl found 32 sites naming CCBot as blocked inside llms.txt while their robots.txt never mentioned it. Crawlers read robots.txt; a policy stated anywhere else is a wish.
• —Buying it as a visibility tactic — paying for an llms.txt inside an “AI SEO” package. Google says Search ignores it, the logs say the retrieval bots barely touch it, and the file takes an hour to write.

What we would do differently: spend that hour, then stop. We keep the file because it costs nothing once it is generated from the CMS, because Lighthouse now checks for it, and because writing one factual line per important page is a discipline that improves the pages themselves. We do not bill for it. On the AI visibility audits we run, the levers we see move citations are crawler access in robots.txt, server-rendered content, clear entity naming and third-party mentions; llms.txt is a footnote in that report, and should be in yours.

Questions readers ask about llms.txt

Does ChatGPT read llms.txt?

OpenAI’s crawler documentation covers GPTBot, OAI-SearchBot and ChatGPT-User and tells site owners to manage them in robots.txt; it does not mention llms.txt as an input. In EZY’s 12-week study of 83 sites, OpenAI crawlers requested llms.txt 7 times and robots.txt 3,990 times. ChatGPT will read your file if a user pastes its URL into a conversation, which is a different thing.

Does Google use llms.txt?

Not for Search. Google’s guide to optimizing for generative AI features says you do not need machine-readable or AI text files to appear in Search and that Google Search ignores llms.txt files. Lighthouse’s experimental Agentic Browsing audit does check for it, as a browser-agent readiness signal, not a ranking one.

Do I need llms-full.txt?

Mainly if you are a documentation site that wants agents to load everything in one request. It is a community convention, not part of the spec, and for a marketing site it mostly duplicates the pages. We publish one anyway because our case studies and service detail are easier to hand to an agent as a single file; for most companies the index alone is enough.

Should I block AI crawlers instead?

Decide per bot, in robots.txt, and separate training from retrieval. Blocking GPTBot or ClaudeBot keeps your content out of training sets; blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those answer engines. Perplexity-User and ChatGPT-User fetch on a user’s request and may ignore robots.txt, by the vendors’ own documentation. llms.txt plays no part in this.

If you are not sure whether AI engines can reach your pages at all, or what they say about you when they can, our free AI visibility audit checks crawler access, rendering, structured data and current citations on a real prompt set. It is the same checklist we use in our own SEO and AEO work, and the llms.txt line on it is near the bottom.