What is llms.txt — and Why Your Website Needs It Right Now
Writen by
nguyen hoang khai
As of 2025, 18% of global search queries now display AI Overviews (AI-generated answer summaries), with that figure reaching 30% in the US. (Source: Rankability, 2025) This means more and more people are getting direct answers from AI without ever clicking through to your website.
The real question is no longer “Does SEO (Search Engine Optimization) still matter?” — it’s: How is AI reading your website, and are you in control of what it says about you?
This is why llms.txt — an emerging technical standard — is being quietly deployed by leading tech companies like Anthropic, Cloudflare, and Stripe since late 2024.
What is llms.txt — and Why It’s Not Just “robots.txt 2.0”
Technically, llms.txt is a plain-text file written in Markdown format, placed at your website’s root directory (yourdomain.com/llms.txt). Its contents form a curated map of your website: a description of your company, a list of your most important pages, your areas of expertise, and trusted references.
But here’s the key distinction: llms.txt has a fundamentally different purpose than robots.txt:
- robots.txt — gives instructions to web crawlers (automated bots that scan websites): “you can go here, stay out of there.” It’s an access control mechanism.
- llms.txt — guides LLMs (Large Language Models like ChatGPT, Claude, Perplexity): “here’s what matters most about us — prioritize understanding this.” It’s a semantic communication layer.
In simpler terms: if robots.txt is the security guard deciding who can enter the building, then llms.txt is the internal map that directs VIP guests straight to the most important rooms — instead of letting them wander through every corridor on their own.
llms.txt doesn’t help you “block” AI — it helps you direct what AI says about you.
Why LLMs Need This File — the Real Technical Problem
Modern LLMs don’t crawl entire websites the way Google’s bot does. When answering a question, they typically rely on one of two mechanisms: knowledge already baked in during training (which may be outdated), or RAG — Retrieval-Augmented Generation (a technique that lets AI pull in real-time information from external sources before generating a response).
The problem with RAG: when an AI agent is gathering information about your company to answer a customer’s question, it only reads whatever pages it can reach — with no guarantee those are the pages you most want to represent you. The result: AI might cite a three-year-old blog post, misrepresent your services, or entirely miss your core differentiators.
llms.txt solves this by providing a verified single source of truth — you tell AI directly: “these are the pages with the most accurate and complete information about us.”
Market Landscape — Adoption Is Growing, But Still Very Early
As of mid-2025, the adoption rate of llms.txt across the entire web sits at just 0.3% — that’s 3 in every 1,000 websites. That might sound low, but for early movers it’s a strong signal: the window for competitive advantage is still wide open.
What’s notable is the growth trajectory: from February to May 2025, the number of websites deploying llms.txt grew by 600% — from 15 to 105 independently verified domains. (Source: llms-txt.io, 2025)
Who’s Leading Adoption?
The first wave of implementers is concentrated clearly in the tech and developer tools space:
- Anthropic — the company behind Claude, deployed llms.txt early as a form of “eating their own cooking”
- Cloudflare — the global network infrastructure provider, also a contributor to the standard’s specification
- Stripe — the payments platform, where accurate technical documentation is mission-critical
- Mintlify — the technical documentation platform, along with fast.ai and numerous other B2B SaaS companies
The pattern is clear: companies with complex technical documentation, or organizations where AI misinformation could cause direct harm, are moving first.
The Real-World Limitations You Should Know
A 2025 analysis of 300,000 domains concluded: no major LLM provider has officially confirmed they use llms.txt during training or retrieval. This means the direct impact right now hasn’t been clearly measured. (Source: SE Ranking, 2025)
However, this isn’t a reason to skip it — it’s a reason to approach it correctly: treat llms.txt as a long-term infrastructure investment, not a quick traffic hack.
llms.txt Within the Bigger Picture: AEO and GEO
To understand llms.txt’s true role, it needs to be placed in the context of two optimization frameworks that are replacing traditional SEO:
- AEO — Answer Engine Optimization: focused on getting your content selected as the direct answer to a specific question. The goal is to appear in featured snippets, voice search responses, and AI-generated summaries.
- GEO — Generative Engine Optimization: focused on shaping how AI describes and recommends your brand in broader conversations — not just when someone asks about you directly.
llms.txt is one of the technical tools that serves both goals: it gives AI clean, structured data to both answer accurately and describe your brand faithfully.
SEO gets you found. AEO/GEO gets you cited. llms.txt is the first step to controlling the narrative when AI tells your story.
What This Means for Engineering and Business Teams in Vietnam
In Vietnam, llms.txt adoption is essentially zero — there’s no dedicated survey data for the local market yet. But that doesn’t mean it’s irrelevant, for one key reason:
Your B2B customers are already using AI to research vendors — even Vietnamese ones. ChatGPT, Perplexity, and Claude are being used to evaluate “is company X reliable?” and “which testing outsource partner is best in Vietnam?” If AI answers inaccurately or incompletely about you, you’ve lost the deal before the conversation even starts.
There’s an additional competitive pressure unique to Vietnam’s tech market: many software companies are competing to be recommended by AI (like ChatGPT or Claude) when international clients ask about outsourcing partners in Southeast Asia. Without a well-structured llms.txt, AI will summarize your company based on whatever content it happens to find — including outdated profiles on LinkedIn or Clutch.
How to Create llms.txt — A Practical Guide
A basic llms.txt file isn’t complicated. The standard structure, according to the specification proposed by Jeremy Howard (the standard’s originator), includes:
- Title section (H1): Company name and a concise 1-2 sentence description of the organization
- Overview section (blockquote): A more detailed explanation of the company, products/services, and target customers
- Key documentation list: URLs with short descriptions — prioritizing service pages, case studies, and in-depth blog posts
- Optional resources: Links to FAQ, pricing, and about pages
Beyond the basic file, some websites also create llms-full.txt — an expanded version containing all content already cleaned (stripped HTML, plain text) so AI can read it directly without needing to crawl further.
Tools for automated generation: the LLMagnet plugin for WordPress, or a free generator at llms-txt.io.
Recommendations Based on Where Your Team Is Today
If your website doesn’t have llms.txt yet: This is a 2-4 hour task with zero risk. Create the file, place it at your root domain, list your 10-15 most important pages. A signal you’re on the right track: AI tools like ChatGPT or Perplexity describe your company more accurately after deployment (test by asking them directly).
If you already have llms.txt but haven’t optimized it: Review whether the file accurately reflects your current positioning, and whether it links to case studies and in-depth blog content. Avoid the common trap: copy-pasting your entire sitemap — AI doesn’t need 500 URLs, it needs your 15 best ones.
If you’re a CTO or Engineering Lead: Assign a junior dev to handle implementation in a single session, but have someone who understands company positioning review the content. Technical implementation is easy — the decision about what information to include is what actually matters.
Key Takeaways — 3 Things to Remember
- llms.txt is a defensive investment: You can’t control what AI says about you — but you can provide the right source material for AI to prioritize.
- The best time is now: Adoption is only 0.3% while AI search is growing rapidly — the window for real differentiation is still open.
- It’s one piece, not the whole picture: llms.txt supports AEO/GEO — but high-quality, in-depth, evidence-backed content remains the foundation nothing can replace.
📚 References:
- Is llms.txt Dead? The Current State of Adoption in 2025 — llms-txt.io
- LLMS.txt Adoption Research Report 2025 — Rankability
- LLMs.txt: Why Brands Rely On It and Why It Doesn’t Work — SE Ranking
- How llms.txt Supports Answer Engine Optimization (AEO) — Artversion
- The Ultimate Guide to llms.txt for GEO/AEO — Insidea
