OpenAI Crawlers & ChatGPT Ads: The Complete Robots.txt Guide
OpenAI runs four distinct crawlers, and confusing them costs you in two ways: you can block your own ad review by disallowing OAI-AdsBot, or leak training data by allowing GPTBot when you meant to opt out. This guide maps each user agent to the exact robots.txt rules that keep your ChatGPT Ads reviewable, your AI search visibility high, and your content policy intact.
Primary sources: OpenAI crawler documentation ↗ · openai.com/adsbot.json ↗
The four crawlers and what each one does
OpenAI's crawlers split cleanly into two jobs: surface-level work (search answers and ad validation) and training work (model improvement). Google solves this with a fleet of user agents; OpenAI does the same, and each one is configured separately in robots.txt.
| Crawler | Purpose | IP list |
|---|---|---|
| OAI-SearchBot | Supports ChatGPT search answers; surfaces your content when ChatGPT answers a question | searchbot.json |
| OAI-AdsBot | Reviews and serves landing pages you submit to ChatGPT Ads | adsbot.json |
| GPTBot | Crawls content that can be used to improve or train OpenAI models | gptbot.json |
| ChatGPT-User | Visits pages when a user asks about them; not background crawling | chatgpt-user.json |
Two of these matter for advertisers most days: OAI-AdsBot and OAI-SearchBot. The third, GPTBot, is your training-policy switch. The fourth, ChatGPT-User, is user-initiated and behaves differently from automated crawling.
How robots.txt applies to each bot
OpenAI's crawlers respect the Robots Exclusion Protocol. The user-agent product token in your robots.txt group must match the crawler's product token, exactly the way Googlebot matching works:
- OAI-SearchBot is matched by the token OAI-SearchBot (case-insensitive).
- OAI-AdsBot is matched by OAI-AdsBot.
- GPTBot by GPTBot, and ChatGPT-User by ChatGPT-User.
- Rules are independent: allowing OAI-SearchBot has no effect on GPTBot, and blocking GPTBot does not block ad review.
There is no wildcard “OpenAI” group: if a group says User-agent: OpenAI, none of the four crawlers match it. Advertisers who copy-paste an old AI-crawler blocklist often discover their ad pages were silently disallowed by an unrelated rule — or completely missed — because they target the wrong name.
Ready-to-use configurations
Scenario one — the recommended default for an advertiser who wants ChatGPT Ads review, AI search visibility, but training opt-out:
User-agent: OAI-SearchBot Allow: / User-agent: OAI-AdsBot Allow: / User-agent: GPTBot Disallow: /
Scenario two — full OpenAI opt-out (blocks ad review too). Only choose this if you are not running ChatGPT Ads and do not want AI search citations:
User-agent: OAI-SearchBot Disallow: / User-agent: OAI-AdsBot Disallow: / User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: /
Scenario three — ad pages only (block OpenAI everywhere except the exact landing pages you submit as ads):
User-agent: OAI-SearchBot Disallow: / User-agent: GPTBot Disallow: / User-agent: OAI-AdsBot Allow: /landing/ Allow: /offer/ Disallow: /
Build your own live configuration in the free ChatGPT Ads robots.txt generator and validate how the four bots actually resolve your file with the OpenAI Crawler Checker.
Published IP ranges
OpenAI publishes a JSON file per crawler with the current IP address ranges, so you can build log-style allow lists, firewall rules, or server-rate-limiting without blocking your real customers. The four sources:
OAI-SearchBot — https://openai.com/searchbot.jsonOAI-AdsBot — https://openai.com/adsbot.jsonGPTBot — https://openai.com/gptbot.jsonChatGPT-User — https://openai.com/chatgpt-user.json
Ranges change, so refresh these files on a schedule if you use them in server rules. They pair with the user-agent headers: OAI-AdsBot identifies itself as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot, which our simulator replays exactly.
Why OAI-AdsBot decides ad review
OAI-AdsBot is not a general-purpose crawler. OpenAI sends it only to pages you submit as ads. If the fetch fails — because robots.txt disallows the URL, the page returns noindex, a CDN challenge greets the bot, or the server takes too long — the ad review stalls or the ad is marked unavailable. Advertisers usually discover this only after the “pending review” state never clears.
Before you submit any campaign, run the landing page through the OAI-AdsBot Landing Page Simulator to replay the exact fetch: robots access, HTTP status, redirect chain, noindex, and bot-protection challenges. It flags the three silent killers that a human browser never sees.
Pre-launch checklist
- Verify robots.txt allows OAI-AdsBot for every URL you plan to submit as an ad.
- Verify OAI-SearchBot is allowed too, if you want AI search citations from the campaign page.
- Decide GPTBot policy explicitly — don't inherit a default from a shared template.
- Confirm the URLs return 200 to a plain fetch and contain no noindex tag.
- Make sure no bot-protection challenge intercepts crawler requests on the landing domain.
- Refresh IP-range lists in any server-side rule on a schedule.
- Recheck after robots.txt changes — caching can delay adoption by days.
Frequently Asked Questions
Which OpenAI crawlers exist and what do they do?
OpenAI publishes four crawlers: OAI-SearchBot (surfaces pages in ChatGPT search answers), OAI-AdsBot (reviews and serves ad landing pages submitted to ChatGPT Ads), GPTBot (crawls content for improving or training generative models), and ChatGPT-User (visits pages when a user asks ChatGPT about them, not automated crawling).
Do I need to allow OAI-AdsBot in robots.txt to run ChatGPT Ads?
Yes. OAI-AdsBot is the crawler OpenAI sends to validate landing pages you submit as ads. If robots.txt blocks it, or the page returns noindex, OpenAI may not review it and the ad can fail validation or be delayed. Allow OAI-AdsBot for the URLs you submit, separately from the other bots.
Can I block GPTBot but still allow OAI-SearchBot?
Yes. Every OpenAI crawler is configured independently in robots.txt. You can Disallow GPTBot to opt out of training use while keeping Allow rules for OAI-SearchBot and OAI-AdsBot so your site still appears in ChatGPT search and passes ad validation.
Where can I find the IP ranges OpenAI crawlers use?
OpenAI publishes JSON files with current IP ranges per crawler: openai.com/searchbot.json, openai.com/adsbot.json, openai.com/gptbot.json, and openai.com/chatgpt-user.json. Use these with log-style allow lists or server-side rate limiting rather than blocking by user agent alone.
Does ChatGPT-User obey robots.txt?
ChatGPT-User is not a background crawler — it visits a page when a ChatGPT user explicitly asks about it. OpenAI documents robots.txt as the opt-out mechanism and the crawlers respect robots directives, but for user-initiated access the practical lever is controlling what content is public on the site.
Turn these rules into shipping campaigns
The Ridoway Pixel handles ChatGPT Ads tracking and deduplication, and the crawler tools above pre-flight every landing page before OpenAI reviews it.