llms.txt: What It Is and Whether It Helps

Someone on your team has read that every website now needs an llms.txt file. Maybe an SEO plugin offered to generate one, or a developer mentioned it on a call. You want to know whether it deserves an afternoon, or whether it is another acronym with nothing behind it.

We publish one on airful.io and have read the proposal closely. Our verdict is short. It costs almost nothing, and nobody has shown that it moves rankings or citations.

Quick answer

llms.txt is a plain-text file at the root of a website, written in Markdown, that gives AI agents a short summary of the site and a curated list of the pages that matter most. It is a proposal by Jeremy Howard, revised in August 2026, and no standards body has adopted it. Google says Google Search ignores the file, and the other major assistants have not committed to reading it. Publish one if it is cheap for you, and do not expect it to change how often you are cited.

What the file is

The idea comes from the llms.txt proposal, first published by Jeremy Howard in September 2024 and revised in August 2026. It suggests placing a Markdown file at /llms.txt so an agent can find concise information about a site in one place. Web pages bury their content under navigation and scripts. Turning that back into clean text is imprecise work for a model.

The proposal is written first for software documentation, where coding agents follow the file to find API references and tutorials. It also names other uses, including a business describing its structure and policies. That second case is the one most founders are hearing about.

What goes inside it

A file that follows the proposal contains these parts, in this order:

• An H1 with the name of the site. This is the only required part.
• A blockquote with a short summary of the site.
• Optional paragraphs or lists with more detail, with no headings.
• Sections under H2 headings, each a list of links with optional notes beside them.

A section headed "Optional" holds links an agent can skip when it needs to save space.

The selection matters more than the syntax. A good file reads like a briefing note for someone who has never heard of you. It says who you are and points to the pages that answer the questions buyers actually ask. A dump of every URL on the site defeats the purpose.

How it differs from robots.txt and sitemap.xml

All three files sit at the root of a site and do different jobs.

Google's robots.txt documentation states that a robots.txt file tells search engine crawlers which URLs they can access on your site. It is about permission.

A sitemap.xml file is an inventory. It lists the pages you want indexed, often with a last-modified date, so a search engine can find and recrawl them.

An llms.txt file is a curated guide. It opens with a summary of the business, then lists the pages that matter with a line on what each covers. The proposal argues that a sitemap is a poor substitute, because it is usually too large to fit in a model's working context and rarely points to plain-text versions of pages.

Nothing in llms.txt overrides robots.txt. If your robots.txt blocks a crawler from the whole site, that block covers /llms.txt too.

What is known about who reads it

Google's guide to optimizing for generative AI search is the clearest statement from any major vendor. It says you do not need llms.txt or similar files to appear in Google Search, including its AI features, because Google Search does not use them. It adds that publishing one will neither harm nor help your visibility there.

Google's tooling is less consistent than its search team. The proposal notes that Chrome's Lighthouse checks for an llms.txt file as part of its agentic browsing audits, so a missing file can show up as a flag even though Google Search ignores it.

We could not find a public commitment from the other major assistant makers to read a site's llms.txt when answering a question. Several AI labs publish llms.txt files for their own developer documentation, as the proposal points out. Publishing a file for developers is a different thing from reading yours when a buyer asks who to hire.

We have also seen no public evidence that adding the file changes rankings or citation rates. The posts we have read that claim a lift do not separate the file from everything else the site changed at the same time.

Some tools do fetch it. The proposal lists plugins and agent tools built to find and search llms.txt files, and coding assistants use them to read documentation. We do not know whether any chat assistant uses yours to describe your business.

How we built ours

We generate our llms.txt from the site's own content when the site is built. Nobody edits it by hand. The same content that renders our capability, engagement and industry pages also supplies their titles and summaries to the file, so a new page appears in it on the next deploy.

The file opens with an H1, a one-sentence blockquote about Airful and a few lines of entity facts, such as the canonical URL, the legal entity and the contact address. Then come H2 sections for capabilities, engagements grouped by capability, industries, important pages and journal posts. Most entries are a title, a one-line summary and a URL.

One detail is easy to get wrong. The proposal asks for each list item to be a Markdown link in square brackets, with any description after a colon. A parser written strictly to the specification may skip entries that carry a bare URL, so check yours against the format.

Beside it sits a companion file, llms-full.txt, which holds the plain text of every capability, engagement and industry page, with their FAQs. Ours is about eight times the size of the short file.

The file only summarises pages, so the pages carry the weight. We cover how to write them in The AEO Playbook: How to Get Cited by AI Answer Engines. If the pages are vague, the summary will be vague too.

How to check your own

Start by opening yourdomain.com/llms.txt in a browser. You should see plain text that begins with a single # heading naming your business. If you get your 404 page or a page of HTML, you do not have one, whatever your plugin settings say.

Then read it the way a stranger would. Check that the summary describes what you sell today and that every link opens. Look for retired services and old pricing.

Next, open your robots.txt and confirm it does not block the AI crawlers you want to reach you.

Finally, search your server or CDN logs for requests to /llms.txt. It is the only direct evidence of use you will get. A request shows that something fetched the file. It does not show that the content shaped an answer, so pair the log check with asking assistants directly, as described in How to Check What AI Says About Your Business.

For a wider review of AI visibility, our Answer Engine Optimisation engagement is a one-week audit of how your brand appears when someone asks an AI assistant who to work with. It looks at the pages themselves, which a file like this can only summarise.

Where to start

1. Open /llms.txt on your own domain and write down whether it exists and what it says.
2. If you have none, write a first version by hand with an H1, a two-sentence summary and links to the pages that answer your buyers' most common questions.
3. If your CMS or framework can generate the file, switch to that so it stays current.
4. Once a month, check your robots.txt and your access logs for requests to the file.
5. Put most of your effort into the pages themselves, since the file can only point at them.

Frequently asked questions

What is an llms.txt file?

An llms.txt file is a plain-text file at the root of a website, written in Markdown, that gives AI agents a short summary of the site and a list of its most useful pages. It was proposed in the llmstxt.org specification in September 2024.

Does llms.txt improve rankings or AI citations?

There is no public evidence that it does. Google says Google Search ignores llms.txt files, and the other major assistants have not committed to reading them. Treat it as a low-cost courtesy to agents.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers which URLs they may access, while llms.txt describes what is on the site and where the useful pages are. A site can publish both, and they do not conflict.

How do I check whether my site has an llms.txt file?

Open yourdomain.com/llms.txt in a browser. If it returns plain text that starts with a heading naming your business, you have one. To see whether anything reads it, search your server or CDN logs for requests to that path.