llms.txt — robots.txt
for the AI era.
New standard. Different purpose. Where robots.txt tells crawlers what not to read, llms.txt tells language models what to read — and gives them clean, structured links to do it.
What it is, and why it might be the next sitemap.xml.
A proposed standard from Answer.AI.
Proposed by Jeremy Howard (Answer.AI, fast.ai) in September 2024 at llmstxt.org. The goal: give large language models a single, curated entry point that explains your site without forcing them to render JavaScript, parse navigation, or guess at hierarchy. It is not a W3C standard (yet) — it's a community convention being adopted by docs platforms (Mintlify, Vercel, Cloudflare, Anthropic, Stripe).
Two files at your domain root.
- ·
/llms.txt— the curated index. A short markdown file with site name, one-line description, and links to your most important pages with one-sentence summaries. - ·
/llms-full.txt(optional) — the full body. Concatenated markdown of every page you want LLMs to ingest. Big file, but cheap for a model to consume.
This generator builds the first one. The second is usually generated by your CMS or a build script (we do these for clients).
Markdown, with strict structure.
# Site Name
> One-sentence summary. Tells the model in plain English what this site is.
A free-form paragraph or two of additional detail.
Mention what's in scope, what's out of scope, the audience.
## Docs
- [Quick start](https://example.com/docs/quickstart): How to set up in five minutes.
- [API reference](https://example.com/docs/api): Full endpoint docs with examples.
## Blog
- [Latest post title](https://example.com/blog/slug): One-line summary of what it covers.
## Optional
- [Pricing](https://example.com/pricing): Plans and limits.
Required: the H1 site name, the > blockquote summary. Recommended: H2 sections grouping links by intent. Each link is a markdown link followed by : description. Sections named "Optional" tell the model it can skip them under context pressure.
AI-native search is the new top-of-funnel.
ChatGPT search, Perplexity, Claude with web access, Gemini Deep Research — these tools are increasingly how people find vendors, products, and answers. They don't render JS. They struggle with deep navigation. They cite cleanly when given clean entry points. A good llms.txt is the difference between "we appeared in 3% of relevant AI answers" and "we appeared in 40%."
It's also forward-defensive: if the standard takes off, sites that adopted early get the link-equity head-start. If it doesn't, you've lost an afternoon writing a clean site index — which has secondary uses anyway (newsletter, sales onboarding, internal docs).
Five things to do, three to avoid.
- ·Link to canonical URLs. Absolute, not relative.
- ·Use HTTPS. Always.
- ·Group by intent (Docs, Blog, About, Pricing).
- ·Write descriptions for humans first — models are trained on human prose.
- ·Re-generate when your site's IA changes.
- ·List every URL on your site — that's what the sitemap is for.
- ·Stuff keywords into descriptions. Models notice.
- ·Forget the H1 + blockquote — without them, parsers may reject the file.
Build your llms.txt.
Fill the basics, add as many sections as you need. Each section can hold any number of links. The preview updates as you type.
Save as llms.txt at the root of your domain. Verify at yoursite.com/llms.txt.
Found this useful? Share it — keeps the tool free + updated.
No login, no email gate. We just like knowing it’s being used.
Other free tools that fit the same workflow.
Robots.txt Generator
Build a valid robots.txt with presets for common policies — and a plain-English explainer of every directive.
AI Brand Mention Checker
Asks Claude, ChatGPT, Gemini & DeepSeek 5 buying-intent prompts. Scores AI visibility 0–100 (AIVS). Email-gated.
We build the full llms-full.txt
at build time.
Auto-concatenated from your CMS, regenerated on every deploy, validated against the spec. Plus AI-search analytics so you know which queries cite you. That's a Foreground build.