llms.txt

In short

• llms.txt is a plain Markdown guide to your site, written for language models instead of browsers.
• It has a title, a one-paragraph summary, and lists of links to the most useful pages.
• It is a community proposal from 2024, not a formal standard, and support varies by tool.
• It takes little effort to add, so treat it as a useful extra alongside good pages, not a substitute for them.

Also called: llms-full.txt, LLMs.txt, llms.txt file, Markdown for LLMs.

Where llms.txt came from

Jeremy Howard of Answer.AI proposed llms.txt in September 2024. The problem it addresses is simple. Web pages are built for people in browsers, full of navigation, scripts, and layout. A language model reading a site has a limited context window and does better with clean text and a clear map of what is where.

The proposal suggests two things: a file at /llms.txt that summarizes the site and links to key content, and Markdown versions of important pages, available at the same address with .md added. Some sites also publish an llms-full.txt with all of their content in one file.

What goes in the file

Each link can carry a short note saying what the page contains. A section titled Optional marks links that a tool can skip when it needs to save space. The format is ordinary Markdown, so the file is also readable by people.

Choie's own file follows this shape: a one-paragraph summary that includes its prices in US dollars, then link sections for services, answers, work, guides, and the glossary, with the sitemap and legal pages under Optional.

Who uses it

Adoption has been strongest among documentation sites and developer tools, where programmers point AI coding assistants at a library's docs. In that setting, a clean Markdown summary saves tokens and improves answers.

Choie's builds use AI in this setting too. It drafts implementation from a decided design and proposes tests, and a senior person reads every line before it ships. See AI-accelerated development.

For general AI search, the picture is less clear. It is a proposal rather than an adopted standard, and AI search products have not made clear commitments to read it. The main route into AI answers is still the same as in search: pages that are crawlable, clear, and useful. That is the work of answer engine optimization.

Should you add one?

For most product and company websites, yes, as long as it stays small. A useful file takes an hour or two to write. It should say in plain words what the company does, who it is for, and what it costs if prices are public, then link to the pages a model would need to answer questions about you.

• Generate it from your content. On a custom website or a site run from a headless CMS, build the file from the same data as the pages.
• Keep facts identical. The prices and claims in the file should match the site word for word.
• Link to Markdown where you can. Clean .md versions of key pages are easier for models to read than HTML.
• Do not hide anything in it. Content meant only for machines, or that differs from what people see, erodes trust.

Choie publishes both files. The llms.txt at choie.co links Markdown versions of the site's pages, and llms-full.txt holds the whole site, glossary included, in one document. Both are built from the same data as the pages, using Next.js routes.

Common mistakes

• Treating it as a ranking trick. It will not make up for thin or unclear pages.
• Listing everything. A file with hundreds of links loses the point. Pick the pages that answer real questions.
• Letting it go stale. Old prices in llms.txt can end up repeated in AI answers.
• Confusing it with robots.txt. llms.txt does not grant or block access. Crawling rules stay in robots.txt.

Common questions