llms.txt in Production: What Broke, What Worked, How I Monitor It

The file that failed silently

In July I ran Lighthouse’s agentic-browsing audits against my own site. The score came back 67, with this line in the report:

File does not appear to contain any links.

The file existed. It returned 200 with the right content type. It described the site in clean prose, and a human reviewer would have approved it. The parser scored it as empty because I had written the URLs as plain text. The spec is built around markdown links, and a machine reading https://damien-martire.com inside a sentence sees a string, not a destination.

I rewrote the file to match llmstxt.org: one H1 with the site name, a blockquote summary, then sections of [title](url) links with a one-line description each. Six links in total, covering the English and French home pages, two owned properties, LinkedIn, and an email. The agentic score went from 67 to 100.

The takeaway: llms.txt has two audiences, and the machine audience does not read prose. Validate the file with a parser, not with your eyes.

The spec fits on an index card

Four rules cover it:

1. One # H1 with the site or project name. Required, and the only required element.
2. A > blockquote right after it, summarizing the site in a sentence or two.
3. ## H2 sections containing markdown link lists: [page title](url): what this page answers.
4. An ## Optional section for secondary links that an AI can skip when short on context.

Serve it at /llms.txt as text/plain. That is the whole standard.

Shipping it on WordPress without a plugin

Both of my sites serve the file from a PHP snippet rather than a plugin. The snippet hooks early in the request cycle, matches /llms.txt, sends a Content-Type: text/plain; charset=utf-8 header, prints the content, and exits before WordPress builds a page. The code lives in Code Snippets and a mirror copy sits in the project folder, so the file survives theme changes and I can diff it after any edit.

On the portfolio the content is a static string. On the catalog the snippet builds the file from live data and caches the result server-side for six hours, because regenerating a summary of a 5,200-page site on each request would be a self-inflicted denial of service.

The 5,200-page problem

A one-page portfolio needs six links. A programmatic catalog forces an editorial decision, and the wrong decision is a sitemap dump.

An AI agent reading your llms.txt spends tokens on it. A thousand URLs would burn the reader’s budget before it reaches your best content. On the catalog I list entry points instead: the two catalog hubs (French and English), the style hub, the blog indexes, and a handful of pillar guides. If a page would not appear in a two-minute tour you give a first-time visitor, it stays out of the file.

The sitemap already handles exhaustive discovery. llms.txt answers a different question: where should a reader with thirty seconds start.

Treat it like production, not like a launch task

Files rot. A plugin update, a cache rule, a slug change, and your endpoint returns a themed 404 that no human notices, because no human visits /llms.txt.

My monitoring dashboard probes every site daily and shows three things on each site’s health card: the endpoint returns 200, the file starts with an H1, and the markdown link count. The probe is thirty lines of code, needs no credentials, and it caught my own conformity gap once already. If you ship llms.txt without a check on it, you have shipped a file that will break in silence at some point between now and whenever an AI system starts reading it.

Does any AI system read it today?

The honest answer, as I write this in July 2026: no. Not on any schedule I can find evidence for. A 48-day server-log study (February to March 2026, 12,099 bot requests across 19 crawlers including GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot) recorded zero requests for /llms.txt; the only visitor that checked the file was a web-analytics company. No major provider has publicly committed to fetching it. Anyone selling llms.txt as a ranking lever is ahead of the evidence.

I ship it anyway, for three reasons. The cost rounds to an hour. Audit tooling has started checking for it, and Lighthouse scoring agentic readiness tells you where the ecosystem is heading; tooling checks precede platform adoption. And it forces the editorial exercise of stating what your site is in twenty lines, which improves your thinking even if no crawler ever reads the result.

I also measure instead of hoping. Before shipping, I recorded what ChatGPT and Perplexity say about my name and my sites. Thirty days after shipping, I rerun the same prompts and diff the answers. If the file moves nothing, I will write that too.

FAQ

Do I need llms.txt if I already have a sitemap?

They do different jobs. A sitemap is an exhaustive inventory for crawlers. llms.txt is a curated orientation page for readers with a token budget, written in markdown so machines can follow the links.

What goes in it on a large site?

Entry points: hubs, indexes, pillar pages. Aim for the set of links you would show a first-time visitor, and keep secondary material under an ## Optional section.

How do I know mine is valid?

Run an agentic audit (Lighthouse includes one now) or any llms.txt parser. The two failures I see most: no H1, and URLs pasted as plain text instead of markdown links.

Both of my files are live at damien-martire.com/llms.txt and beer-galaxy.com/llms.txt. Read them, steal the structure.