llms.txt generator

llms.txt generator

Enter your website address: the tool reads your robots.txt, your sitemap and up to 30 pages (title, meta description), then writes an llms.txt in the llmstxt.org format and a robots.txt block for AI crawlers. Free, no sign-up, nothing is stored.

Example: https://llmstxt.org — only public sites are read; our crawler follows your robots.txt.

How the generator works

  1. robots.txt is read first. Our crawler (llms-txt-generator) skips every URL your rules disallow, and stops if robots.txt answers with a server error, as RFC 9309 requires.
  2. Sitemap: the sitemaps declared in robots.txt, otherwise /sitemap.xml. Sitemap indexes and .xml.gz files are supported. No sitemap? Internal links from the home page are followed.
  3. Up to 30 pages are read: the home page first, then pages from every section of the site. For each page: <title>, meta description (or og:description, or the first paragraph), noindex, canonical.
  4. llms.txt is written in the llmstxt.org format: one H1 with the site name, a blockquote summary, H2 sections built from your URL folders (/blog/ → ## Blog), legal pages in an Optional section. The output is checked against the format before you see it.
  5. robots.txt block: 20 official AI crawler tokens (OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, Mistral, Common Crawl) with three policies. Your current User-agent: * rules are copied for allowed crawlers, because a crawler with its own group stops reading *.

llms.txt example

What the generator produces for a small shop site with a blog (shortened):

# Atelier Grès

> Handmade stoneware tableware from Normandy.

## Main pages

- [Atelier Grès](https://shop.example.com/): Handmade stoneware tableware from Normandy.
- [Contact](https://shop.example.com/contact): Opening hours, address and phone.

## Blog

- [Choosing a mug](https://shop.example.com/blog/choosing-a-mug): Capacity, handle, glaze: our criteria.
- [Caring for stoneware](https://shop.example.com/blog/stoneware-care): Hot water, no abrasive products, air dry.

## Optional

- [Legal notice](https://shop.example.com/legal): Publisher, host, contact.

More detail on the format, and on who actually reads it, in What is llms.txt?

FAQ

Will llms.txt improve my rankings or get me cited by ChatGPT?

Nobody can promise that. Google Search says it does not use llms.txt, and no AI company has committed to reading it. It is cheap to publish and may help agents and coding tools; the robots.txt rules are the part with a documented effect. Details and sources: What is llms.txt?

Is anything stored?

No. Your URL is used for one generation and the result is sent back to your browser. No database, no cookies, no analytics. Against abuse, a hashed (never clear) form of your IP is counted for at most one hour. Cloudflare, the host, keeps standard technical logs. Details.

Why 30 pages?

The generator runs in a free, short-lived serverless function with a budget of 45 requests and 25 seconds. 30 pages covers most small sites. For larger sites the result is a sample of every section, which you should edit.

Which sites are refused?

Private, local and reserved addresses (for example 127.0.0.1, 10.x, 192.168.x, 169.254.169.254, [::1]), host names like localhost, non-standard ports and anything other than http/https. This protects the service against server-side request forgery.

My site is a single-page app. Will it work?

Only if titles and descriptions are in the HTML the server sends. The generator does not run JavaScript.

How do I block our crawler?

Add User-agent: llms-txt-generator / Disallow: / to your robots.txt. See the crawler details.