llms.txt Explained: The New Standard for AI Crawlers (2026)

llms.txt is a plain-text Markdown file you place at the root of your domain — yourdomain.com/llms.txt — that gives AI systems a concise, curated overview of your site and links to your most important pages. Think of it as a hand-written table of contents written for large language models rather than for search-engine crawlers. It was proposed at llmstxt.org in response to a simple problem: AI assistants have limited context windows and struggle to figure out what matters on a sprawling website.

This guide explains what llms.txt actually does, how it differs from the files you already have, whether it's worth your time, and exactly how to write one.

Table of contents

• What llms.txt is (and isn't)
• llms.txt vs robots.txt vs sitemap.xml
• Does llms.txt actually help?
• What goes in the file
• How to create your llms.txt
• llms-full.txt and /pricing.md
• Common mistakes
• FAQ
• The bottom line

What llms.txt is (and isn't)

llms.txt is a curation file, not an access-control file. It doesn't grant or block anything. Instead, it hands an AI a short, opinionated map: here's what this site is, and here are the pages worth reading first. The spec is deliberately simple — an H1 with your site name, a blockquote summary, then Markdown sections of links with brief descriptions.

The point is signal-to-noise. A crawler landing on your homepage has to infer your structure from navigation, footers, and dozens of links. An llms.txt file skips that guesswork and says, in plain language, "these ten pages explain everything." For a product site, that's usually your homepage, pricing, docs, API reference, and a few cornerstone articles.

llms.txt vs robots.txt vs sitemap.xml

These three files are easy to confuse because they all live at your domain root and all talk to automated visitors. They do completely different jobs.

The short version: robots.txt controls access, the sitemap lists everything, and llms.txt highlights what matters. They're complementary. Adding llms.txt doesn't replace your sitemap — you still need a complete, current sitemap so search engines (and the crawlers behind AI answer engines) can discover every page. If your sitemap tends to fall out of date, that's worth fixing first; see why your sitemap is out of date.

Does llms.txt actually help?

Here's the honest answer: there is no confirmed ranking or citation signal from llms.txt yet. As of 2026, major AI providers have not publicly committed to reading the file, and there's no published study proving it changes how often you get cited.

So why bother? Three reasons:

• It's nearly free. Writing a good llms.txt takes an hour. The downside risk is zero.
• It's an emerging protocol layer. This is roughly where schema.org was in its early days — unproven, then gradually adopted until it became table stakes. Being early costs little.
• It helps agents that do read it today. Some developer tools, retrieval pipelines, and custom agents already fetch llms.txt to orient themselves. If an autonomous agent is evaluating your product, a clean llms.txt gives it accurate facts instead of guesses.

Treat llms.txt as insurance and hygiene, not a growth lever. The things that genuinely move AI visibility are structure, authority, and machine-readable markup on your actual pages — covered in our guide on getting cited by AI search engines.

What goes in the file

The format is Markdown, and the spec is loose. A solid, widely-followed structure looks like this:

# Your Product Name

> One-sentence summary of what your site or product does.

A short paragraph of extra context — who it's for, what makes it
different, the core capability.

## Product
- [Home](https://example.com/): what it is and how it works
- [Pricing](https://example.com/pricing): plans and limits
- [Features](https://example.com/features): full feature list

## Documentation
- [Docs](https://example.com/docs): getting started
- [API](https://example.com/docs/api): API reference

## Resources
- [Blog](https://example.com/blog): guides and how-tos

Keep every link absolute, keep descriptions short and factual, and only include pages you'd genuinely want an AI to read. A bloated llms.txt listing every URL defeats the purpose — that's what your sitemap is for.

How to create your llms.txt

1. Draft the summary. Write one blockquote sentence and one short paragraph describing your site in plain language. No marketing fluff — agents parse facts.
2. List your cornerstone pages. Group links under a few headings (Product, Docs, Resources). Add a short description after each.
3. Use absolute URLs. Every link must include the full https:// protocol and domain.
4. Serve it at the root as text/plain or text/markdown at https://yourdomain.com/llms.txt. Most frameworks let you add a simple route or a static file.
5. Keep it current. A stale llms.txt pointing at moved or deleted pages is worse than not having one. Review it whenever your site structure changes.

Want a working reference? Indexly publishes its own at indexly.dev/llms.txt — a live example you can copy the structure from.

llms-full.txt and /pricing.md

Two companion conventions are worth knowing:

• llms-full.txt — a larger file containing full page content (not just links), for agents that want everything in one fetch. Useful for documentation-heavy sites.
• /pricing.md — a plain-Markdown pricing page. Autonomous agents evaluating products often skip JavaScript-rendered or "contact sales" pricing. A parseable /pricing.md with concrete numbers and limits means an agent can actually compare you. Indexly serves one at indexly.dev/pricing.md.

Neither is required, but both follow the same principle: make your key facts machine-readable so agents don't have to guess.

Common mistakes

• Treating it as access control. llms.txt doesn't block anything. To control AI crawler access, use robots.txt — see sitemap vs robots.txt.
• Listing every page. Curate. Ten great links beat two hundred.
• Letting it go stale. Broken links in llms.txt actively mislead agents.
• Skipping the real work. llms.txt won't rescue thin, unstructured content. Fix the pages first.
• Relative URLs. Always use absolute links.

FAQ

What is llms.txt?

llms.txt is a plain-text Markdown file placed at the root of a domain (for example example.com/llms.txt) that gives AI systems a concise, curated overview of your site: what it is, and links to your most important pages. It's a proposed standard from llmstxt.org designed to help large language models find and understand your key content without crawling every page.

Is llms.txt the same as robots.txt or a sitemap?

No. robots.txt tells crawlers which URLs they may access, and an XML sitemap lists every URL you want indexed. llms.txt is different: it's a human-written summary and a shortlist of your best pages, aimed at AI assistants rather than search-engine crawlers. Keep all three — they serve different jobs.

Does llms.txt improve my rankings or AI citations?

There's no confirmed ranking signal from llms.txt yet, and major AI providers haven't officially committed to reading it. Treat it as a low-cost, emerging protocol layer — similar to early schema.org. It can't hurt, it's trivial to add, and it may help agents parse your product accurately today and more so as adoption grows.

How do I create an llms.txt file?

Create a Markdown file with an H1 for your site name, a blockquote summary, then sections of links to your key pages with short descriptions. Serve it at yourdomain.com/llms.txt as plain text. Keep it current — a stale llms.txt is worse than none. Indexly publishes one at indexly.dev/llms.txt as a working example.

The bottom line

llms.txt is a small, cheap, forward-looking addition: a curated Markdown map of your site for AI assistants. It doesn't replace robots.txt or your sitemap, and it won't fix weak content — but it costs an hour, carries no downside, and positions you for a web where AI answer engines matter more each year.

Start with the fundamentals, though. An AI can only cite pages it can find, and it finds them through a complete, current sitemap. Indexly crawls your live site, generates that sitemap, hosts it on a permanent URL, and keeps it fresh automatically — so every page is discoverable by search engines and the crawlers behind AI answers alike. Start free at indexly.dev.