AI crawler (AI bots)

An AI crawler is software that reads websites for an AI company, either to collect text to train its models or to find pages its assistant can cite in answers.

Reviewed by Calum Peak, CTO Last reviewed

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
  • GPTBot Collects text to train OpenAI's models: blocked
  • OAI-SearchBot Finds pages ChatGPT can cite: allowed
  • Googlebot Google Search and AI Overviews: allowed
# Java Roasters
> Manchester roaster. Office coffee delivery
> and wholesale beans for cafés.

- [Office coffee](/office-coffee)
- [Wholesale](/wholesale)
  • Google Search Says it doesn't use the file
  • Other AI search tools None of the major ones has said it reads it
Java Roasters' robots.txt, and what each rule means for each crawler.

What it means for your advertising

Different crawlers do different jobs. OpenAI's GPTBot collects text for training, while OAI-SearchBot finds the pages ChatGPT can show and cite in its answers. Google uses a separate setting, Google-Extended, to control whether your content trains its Gemini models; it has no effect on Google Search, including AI Overviews.

Your site's robots.txt file can allow or block each crawler by name. Blocking a training crawler keeps your pages out of future training; blocking a search crawler can keep you out of that assistant's answers. You can block one and allow the other.

Some website builders and security services block AI crawlers by default. If you want to be mentioned in AI answers (GEO), check what yours does.

An example

Java Roasters' robots.txt blocks GPTBot and allows OAI-SearchBot. Its pages stay out of OpenAI's training, and ChatGPT can still find and cite its office coffee guide.

Java Roasters is a fictional business, and its figures are illustrative.

How Digital Gurus helps

When you advertise with us, our experts can check whether your site lets AI search crawlers in, as part of reviewing the site your ads send people to, and suggest what to change. We run paid advertising, so this is advice alongside your campaign rather than a separate service.

Are you ready to grow your business?

No retainers. No lock-ins. Campaigns when you need them.

Book a strategy call Start in the Guru

Strategy calls run for 30 minutes — bring your business and your goal, we will do the rest.