LLMs.txt and AI Search Visibility: Facts vs. Fiction

In September 2024, Jeremy Howard published a short proposal on the Answer.AI blog. The idea was elegant: create a lightweight markdown file at your website root that gives large language models a concise, structured guide to your most important content. Call it llms.txt. The concept spread fast through developer communities and SEO circles.

By October 2025, over 844,000 websites had implemented it according to BuiltWith's tracking. Anthropic, Cloudflare, and Stripe all added llms.txt files to their own sites. Yoast SEO automated its generation. An entire cottage industry of llms.txt generators, validators, and directories emerged within months.

There is just one uncomfortable piece of data sitting underneath all of this activity. Not a single major AI platform has officially confirmed that they read or use llms.txt files in any meaningful way.

This post is the honest version of the llms.txt story: what it is, what it actually does, what the data shows, and what the expert disagreement tells us about the broader question of AI search visibility in 2026.

What llms.txt Actually Is (The Technical Reality)

Before evaluating whether llms.txt works, it helps to understand precisely what it does and does not do. The confusion in most coverage stems from conflating what the file was designed to accomplish with what people hope it accomplishes.

llms.txt is a plain text file written in Markdown, placed at the root of your website at yoursite.com/llms.txt. It follows a specific structure proposed by Jeremy Howard. Here is what a correctly formatted file looks like:

The file was explicitly designed for one specific purpose: inference-time use, meaning when an AI model is actively answering a question and needs to fetch information from your site in real time. The idea is that instead of parsing your full HTML, with navigation, ads, cookie banners, and JavaScript clutter, the model can read a clean, structured summary and decide which of your pages are worth fetching.

Critically, the proposal also suggests adding .md versions of individual pages at the same URL. So yoursite.com/docs.html.md would be a clean Markdown version of your documentation that AI models can read without parsing HTML.

The Case For llms.txt (What Proponents Actually Have Evidence For)

The story is not purely negative. There are real, documented cases where llms.txt provides genuine value, and dismissing the concept entirely would be as inaccurate as overselling it.

AI coding tools and developer agents read it

This is the strongest evidence for llms.txt's practical utility. If your audience includes developers using Cursor, Claude Code, GitHub Copilot, or similar AI-assisted coding tools, those agents actively fetch and read documentation files. When a developer asks an AI coding assistant to help them implement your API, the agent looks for structured documentation. An llms.txt file pointing to clean Markdown versions of your API docs measurably improves how those agents reason about your product.

This is why Anthropic, Stripe, and Cloudflare all implemented it. Their primary audience is developers. Their content is technical documentation. llms.txt is genuinely useful in that specific context.

Anthropic specifically requested it from Mintlify

AI bots are crawling llms.txt files

Data from Profound, a GEO tracking company, shows that bots from Microsoft and OpenAI are actively fetching both llms.txt and llms-full.txt files. This is confirmed by server log data across multiple domains. The crawling is real. The question is whether that crawling translates into meaningful usage, and on that question, evidence is absent.

The exercise itself has value

Writing an llms.txt file forces you to answer a question that most content teams have never explicitly addressed: what would you want an AI to cite about your product? Curating a one-paragraph summary and a 30-link index of your most important pages requires editorial decisions that clarify your own content hierarchy. That clarity has value independent of whether AI models read the file.

The Case Against the Hype (What the Data Actually Shows)

This is where the research gets uncomfortable for the llms.txt enthusiast community.

Mueller's statement is direct. Google, which operates both traditional search and AI-powered search features including AI Overviews, has not integrated llms.txt into any of its systems. He did not say "we might use it eventually." He said no AI system currently uses it.

OpenAI has not announced that ChatGPT or GPTBot parses or acts on llms.txt content. Anthropic has not confirmed that Claude's systems reference it during conversations, despite publishing their own llms.txt for their documentation. Perplexity and Meta have maintained silence.

Facts vs. Fiction: The Full Breakdown

How llms.txt Compares to Existing Standards

Part of the confusion around llms.txt comes from drawing imprecise analogies to existing web standards. Here is how the comparison actually holds up.

The robots.txt comparison is the most misleading analogy in the current llms.txt discourse. robots.txt works because search engines and AI crawlers are programmed to look for it, respect it, and act on its directives. That programmatic relationship does not currently exist for llms.txt. No AI platform has committed to treating it as authoritative guidance in the way bots treat robots.txt.

What Actually Moves the AI Visibility Needle Today

If llms.txt is not the lever, what is? This question matters more than the llms.txt debate itself, because AI-generated answers are now a meaningful traffic source that businesses cannot afford to ignore.

robots.txt with explicit AI bot rules

This is the lever that provably moves outcomes today. AI crawlers including GPTBot (OpenAI), ClaudeBot (Anthropic), ChatGPT-User, Claude-User, PerplexityBot, and Google-Extended all respect robots.txt directives. If you want to control whether these bots can crawl and potentially cite your content, robots.txt is the mechanism that actually works. Here is an example of a thoughtful 2026 robots.txt setup:

Content quality and specificity

AI models cite sources that answer questions precisely. Broad, generic content does not get cited. Specific, authoritative answers to clearly defined questions do. The most durable AI visibility strategy is writing content that is the best available answer to a specific query, not content optimized for a technical file that AI systems may or may not read.

Entity coverage and consistent brand mentions

Generative Engine Optimization (GEO) research consistently shows that entities mentioned more frequently and consistently across high-authority sources appear more often in AI-generated responses. Building your brand's presence through PR, citations, and third-party mentions is more reliably effective than file-level optimizations.

Structured data and schema markup

Google has explicitly confirmed that structured data helps AI Overviews understand and present your content accurately. This is an evidence-based lever. Schema markup on FAQs, products, how-tos, and organization entities demonstrably affects how AI-assisted search surfaces your content.

Should You Add an llms.txt File? A Decision Framework

• ✓

  Your site is developer-focused with technical documentation. This is the clear yes case. Developers using AI coding tools will benefit from clean Markdown documentation linked from llms.txt. Stripe, Anthropic, and Cloudflare added it precisely for this reason.

• ✓

  You want to future-proof against AI platforms adopting the standard. Implementation takes 10 to 30 minutes and causes zero harm if unused. As a low-effort hedge, it is reasonable. If major AI platforms eventually adopt it, you are already positioned.

• ✓

  You want the editorial clarity that comes from writing it. Curating your most important content into a structured llms.txt file is a useful exercise regardless of whether AI models read it. It forces content prioritization decisions that benefit your overall content strategy.

• ~

  Your site is a content blog or marketing site. The benefit is speculative and the data does not show measurable impact. Adding it is harmless, but it should not take priority over improving content quality, building backlinks, or improving page speed.

• ✕

  You expect it to boost your ChatGPT or Perplexity citation frequency. The data does not support this expectation. OtterlyAI's 90-day study found 0.1% of AI bot requests reached the llms.txt file. If AI citation frequency is your goal, focus on content quality and entity coverage instead.

• ✕

  You are using it as a training data opt-out. llms.txt has no mechanism for this and was not designed for this purpose. robots.txt with specific User-Agent rules for AI training crawlers is the correct tool for controlling what gets used in training data.

The Broader Context: What GEO Actually Means in 2026

The llms.txt debate is a microcosm of a larger conversation about Generative Engine Optimization (GEO), the emerging practice of optimizing for AI-generated search and answer systems rather than traditional search result pages.

GEO is real. AI-assisted search interfaces from Google, Perplexity, Bing, and others now drive meaningful traffic. Being cited in an AI-generated answer has measurable value. The question is which signals actually influence those citations.

Current research suggests the most reliable factors are: content that directly answers specific questions with authority, consistent entity mentions across third-party sources, technical accessibility for AI crawlers via properly configured robots.txt, and structured data that helps AI systems correctly attribute and represent your content.

llms.txt is a technically sensible idea that may eventually fit into that ecosystem. It is not currently one of the measurably effective factors. The honest position is to implement it if it is low effort for your site architecture, treat it as infrastructure rather than a strategy, and focus your primary AI visibility effort on the factors that have demonstrated impact.