llms.txt is an emerging community convention for placing a concise, curated Markdown map of a site's most important resources at /llms.txt, so LLM-based tools and agents can find the right context more efficiently. It was proposed by Jeremy Howard. It is not a formal web standard and is not governed by a standards body.

Is llms.txt a formal web standard?

No. llms.txt is an emerging community convention proposed by Jeremy Howard. There is no standards body, no official specification, and no compliance process. Individual AI products decide independently whether and how they use it. Treat it as a useful early-adopter practice, not an adopted web standard.

What should go in an llms.txt file?

A project H1 heading, a one-line summary blockquote, an optional context paragraph, and H2-grouped sections of your most important links — each with a brief description. Focus on the pages that matter most for understanding your product or documentation: quickstarts, core API reference, key concept guides, and key product pages. Keep it concise — 10–20 high-signal links is better than 60 links of mixed importance.

Should I add Markdown versions of my documentation pages?

It is a good practice where feasible. Serving clean .md versions of important pages (e.g. /docs.md, /api/reference.md) reduces HTML noise for AI tools that fetch content directly. You can link these Markdown versions from your llms.txt or llms-full.txt to give tools a cleaner loading option. It is not a strict requirement of the llms.txt convention.

Does llms.txt improve AI visibility automatically?

No. Publishing llms.txt does not automatically improve how AI products retrieve, rank, or cite your content. Different AI systems have different retrieval pipelines and most do not currently read llms.txt automatically. What a well-made llms.txt can do is make your documentation cleaner and faster to load for tools and agents that do support the convention. Continue strong SEO, structured data, and content quality alongside it.

Where should I place llms.txt?

At the root of your domain: https://example.com/llms.txt. If you have a documentation subdomain, publish a separate llms.txt there too: https://docs.example.com/llms.txt. Placing it at a non-root path (e.g. /docs/llms.txt) makes it harder for tools to discover automatically, since the convention assumes the root path.

Can a portfolio or SaaS site use llms.txt, or is it only for API documentation?

Any site can use llms.txt. It is particularly valuable for documentation sites and developer tools because those sites have a lot of structured information that agents may need to navigate. But portfolios, SaaS product sites, agency sites, and content-heavy blogs can all benefit from a well-structured llms.txt that tells AI tools what the site is, who it is for, and where the most important content lives.

How many links should I include?

Focus on quality over quantity. A concise llms.txt with 10–20 well-described, high-importance links is more useful to an AI tool than one with 50 links of mixed importance. If you have many important pages, use an ## Optional section for secondary resources and a separate llms-full.txt for comprehensive coverage. The goal is to maximise signal-to-noise for a tool that may only read the file once.

All tools

llms.txt Generator & Checker

Q: How is llms.txt different from robots.txt?

robots.txt is about crawler access expectations — which paths are allowed or disallowed for specific crawlers. llms.txt is about content: it gives a curated index of your most important pages and context. It has no access-control semantics. It cannot block or allow crawlers. The two files serve completely different purposes and should both be maintained independently.

Create and audit an experimental llms.txt file that gives AI tools a cleaner Markdown map of your site, docs, and core context.

Experimental: llms.txt is an emerging community convention, not a formal web standard. Continue using SEO, robots.txt, and structured data best practices alongside it. No AI product guarantees it will be used for retrieval or ranking.

Start from a template

Project details

Sections & links

Optional

Link to llms-full.txt

Add an Optional section entry pointing to /llms-full.txt

Output variant

# My Project

> Official documentation for My Project — the fastest way to build X.

This file gives AI tools and agents a curated map of our most important documentation, API references, and guides. Use it to quickly find the right page rather than crawling the full site.

## Getting started

- [Quickstart](https://example.com/docs/quickstart): Set up and run your first example in under five minutes.
- [Installation](https://example.com/docs/install): Installation instructions for npm, Yarn, and Docker.
- [Authentication](https://example.com/docs/auth): API keys, OAuth flows, and token management.

## Core concepts

- [Architecture overview](https://example.com/docs/architecture): How the system is structured and why.
- [Data model](https://example.com/docs/data-model): Core entities, relationships, and field definitions.

## API reference

- [API reference](https://example.com/docs/api): Full REST API reference with request/response examples.
- [Webhooks](https://example.com/docs/webhooks): Event payloads, signatures, and delivery guarantees.
- [SDKs](https://example.com/docs/sdks): Official client libraries for Node.js, Python, and Go.

## Optional

- [Changelog](https://example.com/changelog): Release history and breaking changes.
- [FAQs](https://example.com/docs/faq): Answers to common questions.
- [Full documentation (Markdown)](https://example.com/llms-full.txt): Complete documentation in a single Markdown file for one-shot context loading.

This tool runs entirely in your browser. All generation and local audit logic runs client-side. Remote fetches (checker mode) are proxied through a server-side route with SSRF protection. No content you enter or fetch is stored.

Overview

When an LLM-based tool or agent needs to understand a website, it usually starts with HTML pages — which are noisy. Navigation, footers, cookie banners, analytics scripts, and decorative markup all compete with the actual content. For documentation sites, SaaS products, and developer tools, this can mean an agent spends significant context budget on structure and repetition rather than substance.

llms.txt is an emerging community convention — proposed by Jeremy Howard — that addresses this by placing a concise, curated Markdown map at a well-known path on your site (/llms.txt). Instead of crawling every page, an AI tool or agent can fetch this one file and get a structured index of your most important resources: what the project does, where the key documentation lives, and which pages matter most for understanding your product.

It is important to be clear about what llms.txt is and is not. It is not a formal web standard. It is not a crawler-permissions file like robots.txt. It does not guarantee improved AI visibility, rankings, or citations — different AI products handle content retrieval in different ways and most do not yet read llms.txt automatically. What it can do, when done well, is make your documentation cleaner to load for tools that do support it, and make your site architecture more legible to agents that fetch content on behalf of users. That is a practical benefit worth having, especially for documentation-heavy and developer-facing sites.

What this tool helps with

Good fit for

Documentation sites and developer toolsgenerate a structured Markdown index of your quickstart, API reference, authentication guides, SDK docs, changelog, and FAQs — the resources that matter most for technical users and agents.
SaaS productsgive AI tools a concise overview of your product, its features, pricing, integrations, security posture, and support — without requiring a crawl of every marketing page.
Personal portfolioscreate a clear map of your bio, selected projects, case studies, writing, and contact — giving agents a quick way to understand who you are and what you do.
Auditing an existing llms.txtpaste your current file or fetch it remotely to check for proposal-aligned structure, missing summary blockquotes, sections without descriptions, overlong link lists, and missing llms-full.txt linkage.
Generating an llms-full.txt companionfor documentation-heavy sites, generate the full-context companion file that agents can use for one-shot context loading when they need the complete picture.
Improving Markdown documentation discoverabilityunderstand which pages to prioritise and how to structure descriptions so that AI tools can navigate your content more efficiently.

Not a replacement for

Treating llms.txt as a crawler permissions or blocking filellms.txt has no access-control mechanism. It is a content map, not a robots.txt equivalent. If you want to control crawler access, use robots.txt and Cloudflare AI Crawl Control.
Expecting guaranteed AI visibility improvementspublishing llms.txt does not automatically improve how AI products retrieve or cite your content. Different AI systems handle this in different ways. Maintain strong SEO, structured data, and content quality alongside it.
Listing every page on your sitethe value of llms.txt is curation and signal density. A file with 80 links has lower signal-to-noise than one with 10 well-described, high-importance links. Keep it concise.
Publishing and then forgetting to maintain ita stale llms.txt with outdated or dead links is worse than no file. Update it when your documentation or product changes significantly.

How to use it

1
Choose a template or start from blank
Pick the template that fits your site — Documentation, SaaS, Portfolio, Agency, or Custom. Templates pre-fill sensible section headings so you can focus on your specific links.
2
Fill in your project name, summary, and context
The project name becomes the H1 heading. The summary becomes the blockquote — keep it to one clear sentence. The optional context paragraph explains what the file is for.
3
Add only your most important links
Edit or replace the template links. Focus on pages that give the most value: quickstarts, core API reference, key concept guides, and key product pages. Add a brief description to each link.
4
Preview raw Markdown and the rendered interpretation
Switch between Raw Markdown and Rendered Preview to see what AI tools will read. Make sure the structure reads clearly in both views.
5
Copy or download llms.txt
Use the copy button or download as llms.txt. Publish at https://yourdomain.com/llms.txt. For documentation subdomains (e.g. docs.example.com), publish a separate llms.txt there too.
6
Optionally generate llms-full.txt and Markdown doc variants
Switch to the llms-full.txt output to get the one-shot context companion. Consider publishing .md versions of important doc pages (e.g. /docs.md) that serve cleaner Markdown to AI tools.

Common mistakes and fixes

Missing H1 heading

The very first line of llms.txt should be a top-level H1 heading identifying the project: # My Project. Without it, the file lacks the anchor that parsers and agents use to identify the resource.

Missing summary blockquote

A blockquote (> one-line description) directly after the H1 gives AI tools an immediate, concise description of the site. It is optional per the proposal but strongly recommended. Keep it to one clear sentence.

Listing too many links

An llms.txt file with 40+ links is harder for a model to use effectively. Focus on the 10–20 most important pages. Move secondary and reference material to llms-full.txt or an ## Optional section.

Links without descriptions

A URL alone gives AI tools little signal about why the page matters. Add a short description after each link using the format: - [Title](URL): Brief description of what this page covers.

Vague or generic summary

"My company's website" is not useful. Be specific: what the product does, who it is for, and what makes it distinct. The summary blockquote is often the first thing an agent reads.

Placing llms.txt at a non-root path

The convention places llms.txt at the site root: https://example.com/llms.txt. Placing it at /docs/llms.txt or /static/llms.txt makes it harder for tools to discover automatically.

Treating llms.txt like robots.txt

llms.txt has no access-control semantics. It cannot block crawlers or grant permissions. If you need to control AI crawler access, use robots.txt directives and Cloudflare AI Crawl Control.

Forgetting llms-full.txt for documentation-heavy sites

For sites with substantial documentation, llms-full.txt provides a one-shot full-context file that agents can fetch when they need the complete picture. Link to it from an ## Optional section in llms.txt.

Publishing llms.txt and not maintaining it

Stale files with dead links, outdated descriptions, or missing new products or sections reduce the signal quality for AI tools. Treat llms.txt maintenance like you treat sitemap.xml maintenance.

Frequently asked questions

AI Crawler Rules Builder & robots.txt Generator URL Indexability Inspector Meta Tag & SERP Previewer HTTP Headers Checker Robots.txt Checker Schema.org JSON-LD Generator All developer tools

What is llms.txt?

llms.txt is an emerging community convention — proposed by Jeremy Howard — for placing a concise, curated Markdown map of a site's most important information at a well-known path: https://yourdomain.com/llms.txt.

The idea is straightforward: HTML pages are noisy for LLM-based tools and agents. A plain-text Markdown file that indexes your key resources — with clear headings, link titles, and brief descriptions — gives an AI tool a much cleaner starting point than crawling your navigation HTML. For documentation sites, developer tools, SaaS products, and developer portfolios, this can meaningfully reduce the effort an agent needs to find the right content.

It is not a formal web standard. No standards body governs it. Individual AI products decide whether and how they read or use the file. Publishing llms.txt is a useful step toward AI-friendly documentation, not a guarantee of any specific outcome.

How is llms.txt different from robots.txt?

Aspect	robots.txt	llms.txt
Purpose	Crawler access expectations — which paths to allow or disallow	Content map — a curated index of important resources with context
Mechanism	Machine-readable directives that crawlers are expected to follow	Plain Markdown — informational only, no access-control semantics
Standard?	Formal specification maintained by robotstxt.org	Emerging community convention — no formal standards body
Access control	Yes — Disallow directives restrict crawling	No — llms.txt has no blocking or permissions mechanism
Audience	Any web crawler or bot	Primarily LLM-based tools and agents that support the convention
Path	/robots.txt (root)	/llms.txt (root, by convention)
Format	robots.txt syntax (User-agent, Disallow, Allow, Sitemap)	Markdown (H1, blockquote, H2 sections, link lists)

What is llms-full.txt?

llms-full.txt is a companion pattern to llms.txt. Where llms.txt is a compact index — a curated map of your most important pages — llms-full.txt is a larger one-shot context file that contains more complete documentation in a single Markdown document.

The distinction matters for how agents use them: an agent doing a quick lookup may fetch llms.txt to orient itself. An agent that needs comprehensive context — for example, to answer detailed implementation questions without follow-up fetches — may prefer to load llms-full.txt in a single request and work from that.

llms.txtCompact index. 10–30 curated links with descriptions. Fast to load.
llms-full.txtFull context. Complete or near-complete documentation in one file. Better for one-shot loading by agents.

For documentation-heavy sites, publishing both is recommended. Link to llms-full.txt from an ## Optional section in your llms.txt so tools that support it can discover the companion file.

How to make your site more AI-friendly without overclaiming

Keep your docs clean and current — Outdated documentation is worse than no documentation for any tool, human or AI.
Prioritise your most important resources — A concise, high-signal llms.txt is more useful than an exhaustive one. Curate ruthlessly.
Write concise, specific link descriptions — "API reference" tells an agent less than "Full REST API reference with request/response examples and authentication details."
Provide Markdown-friendly doc variants where practical — Serving clean .md versions of core pages (e.g. /docs.md) reduces HTML noise for tools that fetch directly.
Publish llms-full.txt for documentation-heavy sites — Let agents that need the full picture load it in one fetch rather than following many individual links.
Continue standard SEO, robots.txt, and structured data practices — llms.txt complements these — it does not replace them. Strong canonical signals, structured data, and robots.txt remain the primary discoverability infrastructure.
Do not promise that llms.txt guarantees AI citations — AI product retrieval pipelines vary widely. A well-made llms.txt is a useful signal, not a ranking factor or citation guarantee.

llms.txt structure cheat sheet

# Project Name                          ← H1: required. First line.

> One-line summary of what this is.     ← Blockquote: strongly recommended.

Optional context paragraph explaining
what this file covers and who it is for.
Keep it brief — 1–3 sentences.

## Section Name                         ← H2: groups related links.

- [Page title](https://url): Description  ← Link with description: recommended.
- [Another page](https://url): Brief desc
- [No-description link](https://url)    ← Link without description: allowed.

## Another Section

- [Link](https://url): Description

## Optional                             ← Secondary / lower-priority resources.

- [Full context](https://example.com/llms-full.txt): Complete docs in one file.
- [Changelog](https://example.com/changelog): Release history.

H1 headingRequired. First line. Identifies the project or site.

Summary blockquoteStrongly recommended. One sentence. Immediately after H1.

Context paragraphOptional. Explains what the file covers and who it is for.

H2 sectionsGroup related links. Use clear, descriptive headings.

Link list itemsFormat: - [Title](URL): Description. Description is optional but strongly recommended.

## Optional sectionFor secondary or lower-priority resources. One optional section per file.

llms-full.txt linkInclude in Optional section if you publish a full-context companion file.

Markdown doc linksConsider linking .md versions of key pages where available for cleaner AI loading.

About the checker / audit

The checker analyzes an llms.txt file for practical quality signals based on the proposal-aligned structure described above. It is a guidance tool, not a standards validator — there is no official specification to validate against.

Root path accessibilityWhether /llms.txt is reachable at the expected path with an appropriate HTTP status.

H1 and blockquotePresence of the required H1 and the recommended summary blockquote.

Section structureWhether H2 sections are present and whether they group links meaningfully.

Link descriptionsWhether links include the recommended brief description after the URL.

Link densityWhether the file has too few links (sparse), too many (noisy), or an appropriate number.

llms-full.txt linkageWhether the file references a companion llms-full.txt for one-shot context loading.

Content-typeWhether the file is served as text/plain, which is conventional for .txt files.

Important: failing checks does not mean the file is unusable. The goal of the audit is to surface practical opportunities to improve clarity and signal for AI tools and human maintainers — not to enforce a rigid binary pass/fail standard.

llms.txt Generator & Checker

Overview

What this tool helps with

How to use it

Common mistakes and fixes

Frequently asked questions

What is llms.txt?

How is llms.txt different from robots.txt?

Is llms.txt a formal web standard?

What should go in an llms.txt file?

What is llms-full.txt?

Should I add Markdown versions of my documentation pages?

Does llms.txt improve AI visibility automatically?

Where should I place llms.txt?

Can a portfolio or SaaS site use llms.txt, or is it only for API documentation?

How many links should I include?

Related

What is llms.txt?

How is llms.txt different from robots.txt?

What is llms-full.txt?

How to make your site more AI-friendly without overclaiming

llms.txt structure cheat sheet

About the checker / audit