What it means for your advertising
Different crawlers do different jobs. OpenAI's GPTBot collects text for training, while OAI-SearchBot finds the pages ChatGPT can show and cite in its answers. Google uses a separate setting, Google-Extended, to control whether your content trains its Gemini models; it has no effect on Google Search, including AI Overviews.
Your site's robots.txt file can allow or block each crawler by name. Blocking a training crawler keeps your pages out of future training; blocking a search crawler can keep you out of that assistant's answers. You can block one and allow the other.
Some website builders and security services block AI crawlers by default. If you want to be mentioned in AI answers (GEO), check what yours does.
An example
Java Roasters' robots.txt blocks GPTBot and allows OAI-SearchBot. Its pages stay out of OpenAI's training, and ChatGPT can still find and cite its office coffee guide.
Java Roasters is a fictional business, and its figures are illustrative.
How Digital Gurus helps
When you advertise with us, our experts can check whether your site lets AI search crawlers in, as part of reviewing the site your ads send people to, and suggest what to change. We run paid advertising, so this is advice alongside your campaign rather than a separate service.