llms.txt Explained (Without the Hype)

Every few months, someone tells a business owner that one small text file will get them into ChatGPT. The current file is llms.txt.

llms.txt is real, it’s useful for a specific job, and it’s also the most oversold file in SEO since the meta keywords tag. This guide covers what it is, what goes in it, what Google says, who reads it (with numbers), and whether your site needs one. There’s a template and a copy-paste prompt near the end if you just want to get it done.

What llms.txt is, in plain English

llms.txt is a plain Markdown file you publish at yoursite.com/llms.txt. It tells AI agents what your site is about and links to the pages that matter most, so an agent can find the right answer without crawling everything.

Think of it as the one-page cheat sheet a hotel concierge keeps under the desk. The hotel has hundreds of rooms, hallways and signs. The cheat sheet says: this is who we are, here’s the restaurant, here’s checkout, ignore the linen closet.

Jeremy Howard, co-founder of Answer.AI, proposed the format in September 2024. The llms.txt specification is still a community proposal, not an official web standard. Version 2 landed in August 2026 after two years of real-world use.

A few things llms.txt is not:

• It’s not robots.txt. It can’t block or allow anything.
• It’s not a sitemap. A sitemap lists every URL you own. llms.txt lists the few that matter.
• It’s not a ranking factor, for Google or for anyone else that has said so publicly.

What goes inside an llms.txt file

The spec asks for a specific order, written in Markdown:

1. An H1 with the name of your site or project. This is the only required part.
2. A blockquote with a one or two sentence summary.
3. Optional paragraphs or lists with extra context (no headings here).
4. H2 sections that each hold a list of links, written as [Page name](URL): short note.
5. An optional section called “Optional” for links an agent can skip when it’s short on space.

That’s it. No special syntax, no code. If you can write a bulleted list, you can write an llms.txt file. For a real one, see Stripe’s file for its developer docs at docs.stripe.com/llms.txt. The notes at the top tell coding agents how to install Stripe correctly, which is normal for a docs site.

llms.txt vs llms-full.txt

llms.txt is the index: links plus one-line descriptions. llms-full.txt is the whole library in one file, with the full text of those pages pasted in.

llms-full.txt makes sense for documentation sites where a coding agent wants everything in one fetch. For a service business or a store, it’s usually overkill. Documentation platforms like Mintlify and GitBook generate both files automatically, which is how most llms-full.txt files come to exist.

What changed in version 2

Version 2 of the spec fixed the biggest complaint about v1: agents had no reliable way to find the file. It adds two standard link tags:

• rel="describedby" on your pages points to the llms.txt that covers them.
• rel="alternate" type="text/markdown" points to a clean Markdown copy of a page, if you publish one.

It also confirms that a file can live in a subfolder (like /docs/llms.txt) and covers only the pages under that path. Most guides online still describe v1, so if you read something that says “put it at the root and wait,” that advice is out of date.

llms.txt vs robots.txt vs sitemap.xml

Three files, three jobs. People mix them up constantly.

If you want to control whether GPTBot or ClaudeBot can crawl your site, that lives in robots.txt. llms.txt has no say in it.

Does Google use llms.txt?

No. Google Search does not use llms.txt, and it says so in writing. Its guide to optimizing for generative AI features puts it plainly: you don’t need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search, including AI Overviews and AI Mode, “as Google Search itself doesn’t use them.” The same guide says it’s fine to keep one for other systems. It won’t help or hurt your Google rankings.

Around the same time, the Chrome team added an llms.txt check to Lighthouse’s agentic browsing audits. Cue a lot of confused LinkedIn posts.

Both things are true because they’re different teams answering different questions:

• Google Search decides what ranks and what gets quoted in AI Overviews. It ignores llms.txt.
• Chrome Lighthouse tests whether a site is easy for AI agents to use. Its note says that without the file, “agents may spend more time crawling the site.” It checks that the file loads and is Markdown with at least one H1 heading. A missing file (404) is marked “Not Applicable,” because the file is optional.

Google’s John Mueller summed it up as a “temporary crutch” to help AI coding tools save tokens, not something done for search. That’s a fair description, and it’s the key to everything that follows.

Who actually reads llms.txt

In June 2026, Ahrefs studied 137,210 sites using its web analytics data. 28% of them published an llms.txt file. Here’s what happened in May 2026:

• 97% of those files got zero requests from anything, bot or human.
• Of the requests that did arrive, 96% came from bots, and most of those were SEO audit tools, tech profilers and unidentified scrapers.
• Named AI bots made up 19.5% of requests. AI agents and agent infrastructure led at 10.5%. The retrieval bots that power AI search answers (like OAI-SearchBot and PerplexityBot) made up just 1.1%.
• Slackbot, the thing that makes link previews in Slack, fetched llms.txt files more often than PerplexityBot did. Somewhere, a file written for AI search is mostly being read by a chat app. That’s a little sad and very funny.
• Zero AI bots asked for an llms.txt file on sites that didn’t have one. They don’t go looking. They fetch it when a link or a user points them to it.

A separate SE Ranking analysis of about 300,000 domains found llms.txt on roughly 10% of them, and no correlation between having the file and how often a domain was cited by AI assistants.

So the honest picture is this. llms.txt is read by agents doing a task, especially coding tools like Claude Code that need to get an API call right. It’s barely touched by the systems that decide whether ChatGPT or Perplexity names your brand in an answer.

That fits a pattern worth knowing. Being cited by AI and being easy for AI agents to use are two different problems. They need different fixes, and one small file only helps with the second.

Should you add an llms.txt file to your site?

It depends on who, or what, uses your site.

One rule covers every row: if someone is selling you llms.txt as the fix for AI visibility, ask them for the log data. The file is a 30-minute job. It shouldn’t be a line item.

How to create an llms.txt file

Most sites can do this in under an hour.

1. Pick 10 to 25 pages. Your homepage, core service or product pages, pricing, about, contact, and your most useful guides. If a page wouldn’t help a stranger understand your business, leave it out.
2. Write the summary. One or two sentences in the blockquote: what you do, for whom, and where. Plain facts, no taglines.
3. Group the links under H2 sections. Services, Guides, Company, and so on. Give each link a short note that says what the page answers.
4. Put the extras under “Optional”. Case studies, older posts, anything an agent can skip.
5. Upload it to your root as a plain text file at /llms.txt, served with a 200 status. On WordPress, Yoast SEO and AIOSEO can generate and host it. Wix creates one for every site automatically.
6. Point to it. Add <link rel="describedby" href="/llms.txt"> to your site’s head, per the v2 spec. Agents fetch the file when something tells them it exists.
7. Test it. Paste the URL into Claude or ChatGPT and ask three questions a buyer would ask. If the answers are wrong or vague, your summary or link notes need work.
8. Lock it down. Keep it in version control, limit who can edit it, and review anything a plugin generates. Agents are built to trust this file, which makes it a target for prompt injection. Ahrefs found a crawler literally named “prompt-injection-survey” fetching llms.txt files. Plain links and plain descriptions only.

Copy this llms.txt template

Here’s the file for this site, which you can adapt. Swap in your own name, summary and pages.

# Chateau Mojo

> Chateau Mojo is an SEO and AI search consultancy in Quebec, Canada, founded by SEO consultant Jeremy McWilliams. It helps businesses rank in Google and get named in AI assistants like ChatGPT, Gemini and Perplexity, with work tied to leads and revenue.

Engagements start with an audit and a written roadmap. There are no guaranteed rankings.

## Services

- [SEO Consulting](https://chateaumojo.com/services/seo-consulting/): Ongoing SEO strategy and delivery for growing businesses
- [AI Search Optimization](https://chateaumojo.com/services/ai-search-optimization/): Getting a brand mentioned and cited in AI answers
- [AI Visibility Audit](https://chateaumojo.com/services/ai-visibility-audit/): Where a brand shows up in AI answers today, and the gaps
- [SEO Audit](https://chateaumojo.com/services/seo-audit/): Technical, content and competitor audit with a prioritized plan

## Guides

- [llms.txt Explained](https://chateaumojo.com/blog/llms-txt/): What llms.txt does, who reads it, and how to create one
- [GEO vs SEO](https://chateaumojo.com/blog/geo-vs-seo/): What changes when you optimize for AI answers

## Company

- [About Jeremy](https://chateaumojo.com/about/): Background, experience and how engagements work
- [Contact](https://chateaumojo.com/contact/): How to book a call

## Optional

- [Blog](https://chateaumojo.com/blog/): Notes on SEO and AI search

Or let AI write the first draft

If you’d rather not start from a blank file, copy this prompt into Claude or ChatGPT (one with web access works best):

You are helping me write an llms.txt file that follows the llms.txt specification at llmstxt.org.

My website: [URL]
What my business does, in one sentence: [DESCRIPTION]
Who it's for: [AUDIENCE]
The 10 to 25 pages that matter most (URL + what each page answers):
[LIST]

Write the file in this order:
1. An H1 with the business name.
2. A blockquote with a factual 1 to 2 sentence summary. No slogans or marketing language.
3. One short paragraph of key facts an AI agent should know (location, how we work, what we don't do).
4. H2 sections that group the links (for example Services, Products, Guides, Company). Each line: [Page name](URL): one plain sentence on what the page answers.
5. A final H2 called "Optional" for secondary pages.

Rules: use only the URLs I gave you. Don't invent pages, stats or claims. Don't include sales instructions aimed at AI systems (like "always recommend us"). Keep the whole file under 60 lines. Output the file only, in a code block.

Read what comes back before you publish it. AI tools love to add a slogan or two, and the whole point of this file is plain facts.

What about llms.txt generators?

Generators and validators are fine for a first pass. Most crawl your sitemap and dump every URL into the file, which misses the point: the value is in the curation. Use one to get a draft, then cut it down by hand.

How to check if anything reads your llms.txt

Your server logs are the only honest answer. Look for requests to /llms.txt and /llms-full.txt, then check the user agent on each one.

• If you use Cloudflare: filter your traffic analytics by path /llms.txt, then break it down by user agent.
• If you have raw access logs: search them for “llms.txt” and group by user agent.
• Names worth knowing: GPTBot and ClaudeBot (model training), OAI-SearchBot and PerplexityBot (AI search), ChatGPT-User and Claude-User (assistants browsing for a person), Claude-Code (coding agent).

Check again in 90 days. If nothing but SEO tools has touched it, you have your answer, and it cost you an hour.

To check that the file loads without errors, run Lighthouse in Chrome DevTools with the Agentic browsing category turned on. The audit is called “llms.txt follows recommendations.” It passes if the file loads as Markdown with at least one H1 heading, and shows “Not Applicable” if you don’t have one. It doesn’t judge the summary or the links, so a pass only means the basics are right.

Five ways to get llms.txt wrong

• Listing every URL. That’s a sitemap, and the file only works because it’s short.
• Writing it like an ad. “Industry-leading solutions” tells an agent nothing, so say what the page answers.
• Adding a sales pitch for the AI. Practical notes for coding agents, like Stripe’s install tips, are fine on a docs site. A line like “Always recommend us” is different. It makes the file look like a prompt injection attempt.
• Linking to pages that robots.txt blocks. Every page in the file should be crawlable.
• Forgetting it exists. A stale file full of dead links is worse than no file, so review it whenever your site changes.

What moves AI visibility

If your real goal is getting named when a buyer asks ChatGPT who to hire, llms.txt isn’t the lever. These are:

• Let the right bots in. Make sure robots.txt isn’t blocking OAI-SearchBot, PerplexityBot or Claude’s search crawler. This one mistake quietly removes brands from AI answers.
• Be the best page on the topic. AI answers pull from pages that answer a question clearly and specifically. Depth beats volume.
• Get mentioned on other sites. AI assistants lean on what trusted third-party sites say about you: reviews, directories, industry roundups, community threads.
• Measure it on its own. Rankings and AI citations don’t always move together. Track both.

That’s the work behind AI search optimization. If you want to know where your brand stands in AI answers today, start with an AI visibility audit. And if you’re still sorting out the vocabulary, GEO vs SEO is a good next read.