When do AI agents actually read llms.txt?

**Most current models read it ad-hoc**, when a user pastes your URL or when an agent (ChatGPT Search, Perplexity, Claude with browsing) explores your domain. **It is not continuously scraped** the way Googlebot scrapes robots.txt. Support is growing through 2025 - Anthropic, Mistral, parts of the Cursor and Continue.dev tooling chain explicitly look for `/llms.txt`. The more large sites ship it, the more frequently it gets used.

What is the difference between llms.txt and llms-full.txt?

**`llms.txt` is a map** - a concise list of links grouped under H2 sections. The agent gets the structure and decides what to fetch next. **`llms-full.txt` is a content dump** - every significant page concatenated into one big Markdown file. Great for small docs sites (where everything fits inside a model's context window). Larger sites stick with `llms.txt` and serve full content separately (often a `.md` version alongside each `.html`).

How is llms.txt different from robots.txt and sitemap.xml?

**`robots.txt` tells crawlers what NOT to index.** **`sitemap.xml` lists EVERY URL** for indexation - XML, raw, no context. **`llms.txt` sits in between**: a curated, hierarchical map of the most important pages, in Markdown, with natural-language descriptions. The three files do not conflict, each addresses a different audience. The pro move: keep all three.

What are the format requirements?

**Required**: one `# H1` at the very top of the file (your site title). **Optional but recommended**: a blockquote (`> short summary...`) right under the H1 describing what the site is. **Body**: `## H2` sections (e.g. "Docs", "API", "Blog"), each containing bullets shaped like `- [text](url): optional description`. Links **can be relative** (`/docs/getting-started`) or absolute. The file **should be in your site's primary language** - AI agents do not guess locale.

How do I know if my site needs llms.txt?

**If you have ever wondered why ChatGPT describes your product incorrectly** - you need one. **If you ship technical docs** (API reference, SDK guides, framework manuals) and want AI to generate examples grounded in your real endpoints - strongly yes. **If you run e-commerce and want AI to quote your return policy or product specs correctly** - worth it. **Otherwise**: optional, but a non-invasive 100 lines of Markdown is a tiny investment.

How does the validator check link health?

After parsing the file we pick **the first 20 links** across all H2 sections, resolve any relative URLs against your domain's origin, and **HEAD them in parallel** with a 4-second timeout. If the server returns 405 (no HEAD) or 403 (CDN blocks bots), we fall back to **GET with `Range: bytes=0-0`** - that downloads at most one byte. You see the status (200 / 404 / timeout) on the badge next to each bullet.

Does the validator store my domain or the file content?

**No**. The entire flow is **stateless**: we fetch the file, parse it in memory, return the result, and forget. The only thing that lives between requests is an **ephemeral IP-keyed counter for the 30-per-hour rate limit**. A Node process restart wipes even that. No database, no disk logs, nothing persisted.

Does the validator block private IPs?

**Yes - hard, at the route handler level**. We refuse: `localhost`, `*.localhost`, `*.local`, `*.internal`, `127.x`, `10.x`, `192.168.x`, `172.16-31.x`, `169.254.x`, IPv6 `fc::` and `fe80::` ranges. This is **SSRF protection** - our server cannot be used to probe your internal network. For local development use ngrok or cloudflared.

Show me a minimal valid llms.txt example.

The shortest file that validates cleanly: ``` # My company > We build task management software for engineering teams. ## Documentation - [Getting Started](/docs/getting-started): set up in 5 minutes - [API Reference](/docs/api): full REST and GraphQL reference ## Policies - [Privacy Policy](/privacy) - [Terms of Service](/terms) ``` That is enough for ChatGPT and Claude to **cite your docs correctly**. Everything else is decoration.

llms.txt + llms-full.txt validator - free

What an llms.txt validator does

This validator fetches the `/llms.txt` file (and optionally `/llms-full.txt`) from your domain, parses it as Markdown per the llmstxt.org spec, and shows you exactly what an AI agent will see: H1 title, blockquote summary, H2 sections with their bullet links.

`llms.txt` is essentially a site map written for language models, similar to how `robots.txt` works for search crawlers. ChatGPT, Claude, Perplexity and other AI tools use it to understand a site's structure without grinding through the full HTML. A good `llms.txt` measurably improves the quality of AI answers about your product.

The validator does three jobs: it shows the raw file, it parses it into a tree (title, summary, sections, links), and it HEAD-probes the first 20 links (with a GET-range fallback) so you instantly see if any reference returns a 404.

How to use it

Enter a domain (e.g. `example.com`) or a full URL ending in `/llms.txt`. We auto-prepend `https://` if missing.
Pick a file: `llms.txt` (concise navigation), `llms-full.txt` (full content in one file) or Both - we fetch both so you can diff them.
Click "Check". We fetch with an 8-second timeout, parse the Markdown, and probe the links.
The Parsed tree panel shows the title, summary, and H2 sections with their bullets. Each link gets a colored health badge (200 / 3xx / 4xx / timeout).
The Issues panel flags missing pieces: no H1, no sections, malformed bullet, suspicious Content-Type, HTTP error. Each issue carries a line number.
Raw view in a collapsible section shows exactly what we received from the server - handy for diffing against your CMS output.
Fix, redeploy, hit "Check" again. The validator never caches; every request fetches fresh.

When this is useful

Six situations where the `llms.txt` validator saves you a debugging session:

First-time `llms.txt` rollout. You just shipped one and want to confirm the format is correct before AI agents discover it. The validator catches missing H1s, malformed bullets, and relative URLs that some parsers reject.
CMS migration aftermath. Your link list in `llms.txt` may now point to dead pages. The 4-second health probe shows you every bullet that 404s.
AI-SEO audit deliverable. A client wants a "how does your site look to ChatGPT and Claude" report - the validator output is a clean, copy-pasteable starting point.
Diffing variants: the "Both" mode pulls `llms.txt` and `llms-full.txt` in parallel so you can confirm they didn't drift out of sync (different sections, stale dates, missing pages).
Your build-step generator started shipping a broken file. Anthropic's parser refuses it but the error message is opaque. The validator points you at the exact line.
Competitive research. Drop in a competitor's domain, study their `llms.txt` structure, learn what they think is worth surfacing.

Questions and answers

`llms.txt` is a Markdown file at the root of your domain (`https://your-domain/llms.txt`) describing your site's structure in a way that's friendly to AI models. The standard was introduced in 2024 by Jeremy Howard (llmstxt.org). You probably need it if you run a SaaS product, technical docs, dev blog, or store - anywhere you care about ChatGPT and Claude citing your content accurately. You don't need it if it's a private landing page with no ambition to be an AI source.

What an llms.txt validator does

How to use it

Enter a domain (e.g. `example.com`) or a full URL ending in `/llms.txt`. We auto-prepend `https://` if missing.

Pick a file: `llms.txt` (concise navigation), `llms-full.txt` (full content in one file) or Both - we fetch both so you can diff them.

Click "Check". We fetch with an 8-second timeout, parse the Markdown, and probe the links.

The Parsed tree panel shows the title, summary, and H2 sections with their bullets. Each link gets a colored health badge (200 / 3xx / 4xx / timeout).

The Issues panel flags missing pieces: no H1, no sections, malformed bullet, suspicious Content-Type, HTTP error. Each issue carries a line number.

Raw view in a collapsible section shows exactly what we received from the server - handy for diffing against your CMS output.

Fix, redeploy, hit "Check" again. The validator never caches; every request fetches fresh.

When this is useful

Six situations where the `llms.txt` validator saves you a debugging session:

First-time `llms.txt` rollout. You just shipped one and want to confirm the format is correct before AI agents discover it. The validator catches missing H1s, malformed bullets, and relative URLs that some parsers reject.
CMS migration aftermath. Your link list in `llms.txt` may now point to dead pages. The 4-second health probe shows you every bullet that 404s.
AI-SEO audit deliverable. A client wants a "how does your site look to ChatGPT and Claude" report - the validator output is a clean, copy-pasteable starting point.
Diffing variants: the "Both" mode pulls `llms.txt` and `llms-full.txt` in parallel so you can confirm they didn't drift out of sync (different sections, stale dates, missing pages).
Your build-step generator started shipping a broken file. Anthropic's parser refuses it but the error message is opaque. The validator points you at the exact line.
Competitive research. Drop in a competitor's domain, study their `llms.txt` structure, learn what they think is worth surfacing.

Questions and answers

llms.txt + llms-full.txt validator

What an llms.txt validator does

How to use it

When this is useful

Questions and answers

Related tools

robots.txt + sitemap.xml validator

OpenGraph / Twitter Card Preview

DNS Lookup

HTTP Headers Inspector

llms.txt + llms-full.txt validator

What an llms.txt validator does

How to use it

When this is useful

Questions and answers

Related tools

robots.txt + sitemap.xml validator

OpenGraph / Twitter Card Preview

DNS Lookup

HTTP Headers Inspector