llms.txt

llms.txt is a proposed file that lists a website's key pages in plain text for AI tools to read. None of the major AI search tools has said it uses one.

Reviewed by Calum Peak, CTO Last reviewed

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
  • GPTBot Collects text to train OpenAI's models: blocked
  • OAI-SearchBot Finds pages ChatGPT can cite: allowed
  • Googlebot Google Search and AI Overviews: allowed
# Java Roasters
> Manchester roaster. Office coffee delivery
> and wholesale beans for cafés.

- [Office coffee](/office-coffee)
- [Wholesale](/wholesale)
  • Google Search Says it doesn't use the file
  • Other AI search tools None of the major ones has said it reads it
Two files at the root of Java Roasters' site: robots.txt, which crawlers follow, and llms.txt, which no major AI search tool has said it reads.

What it means for your advertising

The idea, proposed in 2024, is a simple text file at yoursite.com/llms.txt: a short description of the site and links to its most useful pages, written in a format a language model reads easily. It's named after robots.txt, the file that tells crawlers what they may visit.

The difference is that robots.txt is a long-standing standard that crawlers follow, AI crawlers included, while llms.txt is a suggestion. Google says its search doesn't use llms.txt, and that no special files are needed to appear in its AI features. Studies of server logs find AI crawlers rarely even ask for it.

It does no harm, and some developer tools read it for documentation. But it won't get you into AI answers. Clear pages that search engines and AI crawlers can reach are what count (GEO).

An example

Java Roasters' web developer suggests adding an llms.txt file. It takes an hour, and in the months that follow the server logs show no AI search crawler asking for it. The time would have been better spent on the delivery area section of its office coffee page.

Java Roasters is a fictional business, and its figures are illustrative.

How Digital Gurus helps

When you advertise with us, our experts focus their suggestions for your site on what search engines and AI tools are known to use: your content, its structure and whether crawlers can reach it. If you already have an llms.txt file, there's no need to remove it. We run paid advertising, so this is advice alongside your campaign rather than a separate service.

Are you ready to grow your business?

No retainers. No lock-ins. Campaigns when you need them.

Book a strategy call Start in the Guru

Strategy calls run for 30 minutes — bring your business and your goal, we will do the rest.