Is llms.txt worth it? What it does and who reads it
Google Search ignores llms.txt, so it won't lift rankings. It is cheap to add and useful for AI agents and developer docs. Here is when to bother.
An llms.txt file is a Markdown index at the root of a website that points AI tools to the pages worth reading. Google says its search ignores the file, so llms.txt is not a ranking lever. It is still worth an hour if AI agents or developers use your docs, because it hands them a clean map.
That is the short answer. The rest of this piece explains what the file contains, which tools read it, what the search engines have said, and how to decide for your own site.
What llms.txt is
The format was proposed by Jeremy Howard in September 2024, and the spec lives at the llmstxt.org site. Its opening line states the idea: “We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content.”
The reasoning is practical. As the proposal puts it, “Context windows, while larger than they were, are still too small for most websites in their entirety.” A model that has to read your navigation, cookie banner and footer on every page wastes tokens and time. A short, curated index lets it go straight to what matters.
The authors write that they expected llms.txt “would mainly be useful for inference rather than training.” In other words, it is for a tool fetching your site while answering someone’s question, not for a crawler building a dataset.
What goes in the file
The format is plain Markdown with a light structure. Only the first element is required.
- An H1 with the site or project name. This is the one required line.
- A blockquote summary. One or two sentences on what the site is.
- Optional detail. Paragraphs or lists that give context, without headings.
- H2 sections of links. Each entry is a Markdown link, optionally followed by a colon and a short note.
- An “Optional” section. Secondary links an agent can skip when it needs less context.
A small example for a bakery that sells online might look like this:
# Rye and Crumb
> A neighborhood bakery in Portland that ships sourdough and pastries in the US.
## Ordering
- [Shipping and delivery](https://example.com/shipping.md): where we ship and cut-off times
- [Allergens](https://example.com/allergens.md): what each product contains
## Optional
- [Our story](https://example.com/about.md)The file sits at /llms.txt, or under a subpath such as /docs/llms.txt to cover only that section. When several files apply, the spec says agents should use the most specific one.
The spec also suggests offering a clean Markdown copy of each page at the same URL with .md added, such as page.html.md. Many sites also publish an llms-full.txt with the full text of every page in one file. That convention is common, but it is not part of the llmstxt.org spec.
How llms.txt differs from robots.txt and a sitemap
The three files are easy to confuse because they all sit at the root. They do different jobs.
| File | Who it is for | What it does | Affects search rankings |
|---|---|---|---|
| robots.txt | Crawlers | Says which paths a bot may fetch | Indirectly, by allowing or blocking crawling |
| sitemap.xml | Search engines | Lists every indexable URL for discovery | Helps discovery, not ranking |
| llms.txt | AI tools at answer time | Points to a curated set of readable pages | No, per Google |
The llmstxt.org spec says llms.txt “can complement robots.txt,” and argues a sitemap is no substitute. A sitemap often lacks readable versions of pages, leaves out useful external links, and lists more than fits in a context window.
What Google says about llms.txt
Google’s guide to optimizing for generative AI features addresses the file by name. It says creating llms.txt files “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.”
The same guide is broader: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search.” Google puts llms.txt on the same list of myths as chunking pages into small pieces and rewriting text for AI.
That settles the SEO question for Google. If someone sells you llms.txt as a way to appear in AI Overviews, they are contradicting Google’s own documentation. Our guide to what gets cited in AI answers covers what does help.
Who actually reads llms.txt
No major AI search provider we checked says its search crawler uses llms.txt to choose sources. OpenAI’s crawler documentation and Perplexity’s crawler page explain their bots and robots.txt controls, and say nothing about llms.txt affecting what gets shown in answers.
Yet both of those pages link to an llms.txt as the index of their own developer documentation. That is a telling detail. The companies building AI tools publish the file for the audience it serves best: developers and coding agents that need to read documentation.
The Model Context Protocol documentation index is another example. It lists around 360 Markdown pages, and the docs pages tell agents to fetch that index before exploring further. It is a flat list rather than the curated, sectioned file the spec describes, which shows how loosely sites follow the format in practice.
The realistic readers today are:
- Coding assistants and agents that fetch documentation while working, often because a developer pasted the URL.
- Assistants answering a user’s question that decide to fetch a page and find the index useful.
- Developer tools and plugins that build context from llms.txt files, including the documentation plugins the spec lists.
None of that shows up in a rankings report. It shows up as an agent getting your API right on the first try.
When llms.txt is worth adding
The cost is low: one file you can write by hand, or generate from your sitemap and page titles. The question is whether anyone who matters to you will read it.
Add it when:
- You publish developer documentation, an API or an SDK, and people use coding assistants to work with it.
- You run an MCP server or other agent integration, and you want agents to find the setup and tool docs quickly.
- Your site has long help or policy pages that people paste into assistants, such as shipping rules, pricing terms or product specs.
- Your framework can generate it for free, as the VitePress and Docusaurus plugins the spec mentions do.
Skip it, or leave it for later, when:
- Your goal is search traffic or AI Overview citations. Google ignores the file, and no other engine says it uses it for ranking.
- You have a small brochure site with a few pages that any tool can read directly.
- You would have to maintain it by hand and it would drift out of date. A stale index that points to moved pages is worse than none.
How to add llms.txt without creating new problems
If you decide to add one, keep it honest and low-maintenance.
- Generate it from your real pages. Build it from the same source as your sitemap, so new and removed pages update together.
- Curate the top section. Put the ten to twenty pages a newcomer needs first. Move the rest under “Optional.”
- Repeat, don’t replace. Every fact in llms.txt or a Markdown copy should also be on the human page. Content that exists only for machines is the kind of thing Google’s guidance warns against.
- Keep links canonical. Point to the same URLs your pages use, and keep the Markdown copies in sync with the HTML.
- Don’t block it, and don’t count on it. Make it fetchable, then spend your SEO effort on the pages themselves, starting with solid internal links on every article.
What Wavefront does with llms.txt
This site publishes llms.txt, llms-full.txt and a Markdown copy of each article. They repeat the pages; they don’t replace them. We do it because our readers include agents. Wavefront runs a remote MCP server, and the MCP connection docs are exactly the kind of page a coding assistant fetches mid-task.
The article writer doesn’t generate llms.txt files or any other AI-only content for your site. Google’s guidance says they won’t help you rank, so it spends the effort on the article. If agents are part of your audience too, our guide to connecting AI agents with MCP and the developer overview cover that side.