# OAI-SearchBot vs GPTBot: OpenAI's four bots and what each one controls > OAI-SearchBot puts you in ChatGPT search, GPTBot collects training data, ChatGPT-User fetches for users. The robots.txt for search yes, training no. Source: https://seofix.ai/blog/openai-crawlers-gptbot-oai-searchbot Published: 2026-11-09 Updated: 2026-11-02 Author: SEOFix Team Category: AI search Tags: AI agents, Technical SEO, Guides Length: 1,629 words, 8 min read OAI-SearchBot decides whether your pages can appear in ChatGPT search. GPTBot decides whether OpenAI may use your pages to train its models. They are separate robots.txt tokens, and OpenAI says each setting ["is independent of the others"](https://developers.openai.com/api/docs/bots), so the common policy of "show up in ChatGPT, stay out of training" is two lines: allow `OAI-SearchBot`, disallow `GPTBot`. OpenAI also runs ChatGPT-User, which fetches pages when a person asks ChatGPT about them, and OAI-AdsBot, which checks ad landing pages. Neither of those two is controlled the way the first two are. Everything below comes from [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots) and its published IP files, checked on 2 November 2026. For the same comparison across Anthropic, Perplexity, Google and Apple, see our [AI crawler robots.txt guide](https://seofix.ai/blog/block-gptbot-robots-txt). This post stays with OpenAI and goes one level deeper. ## OAI-SearchBot vs GPTBot (and the other two) | User agent | What it does | robots.txt | IP list | | --- | --- | --- | --- | | `OAI-SearchBot` | Surfaces sites "in search results in ChatGPT's search features" | Obeyed. Changes take about 24 hours | [searchbot.json](https://openai.com/searchbot.json) | | `GPTBot` | Crawls content that "may be used in training our generative AI foundation models" | Obeyed | [gptbot.json](https://openai.com/gptbot.json) | | `ChatGPT-User` | Visits a page when a user asks ChatGPT or a Custom GPT a question | "robots.txt rules may not apply" | [chatgpt-user.json](https://openai.com/chatgpt-user.json) | | `OAI-AdsBot` | Checks pages submitted as ads on ChatGPT | Visits only submitted ad pages | [adsbot.json](https://openai.com/adsbot.json) | A few sentences from OpenAI's page change how you should read that table. **Opting out of OAI-SearchBot removes you from ChatGPT search answers, mostly.** OpenAI says such sites ["will not be shown in ChatGPT search answers, though can still appear as navigational links"](https://developers.openai.com/api/docs/bots). A user who asks for your site by name may still get a link. **ChatGPT-User has nothing to do with search eligibility.** OpenAI: ["ChatGPT-User is not used to determine whether content may appear in Search. Please use OAI-SearchBot in robots.txt for managing Search opt outs and automatic crawl."](https://developers.openai.com/api/docs/bots) It is also ["not used for crawling the web in an automatic fashion"](https://developers.openai.com/api/docs/bots). Each request is a person's question. **One crawl may serve both purposes.** ["If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling."](https://developers.openai.com/api/docs/bots) So if you allow both, don't expect two separate streams of requests in your logs. You may see mostly one of them. **OAI-AdsBot data is not used for training.** OpenAI says it ["only visits pages submitted as ads"](https://developers.openai.com/api/docs/bots) and may use landing page content to decide when to show the ad. If you don't buy ChatGPT ads, you shouldn't see it. ## The exact user-agent strings Copied from OpenAI's page. For OAI-SearchBot and GPTBot, OpenAI labels these "example" strings and says the version number may change, so match on the token, never the full string. ``` # OAI-SearchBot Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot # OAI-SearchBot fetching robots.txt Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot # GPTBot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot # ChatGPT-User Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot # OAI-AdsBot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot ``` Three practical consequences: - The OAI-SearchBot string starts with a full Chrome-on-Mac browser string. A WAF rule that looks for "Chrome" to decide "this is a human" lets it through, and a rule that blocks old Chrome versions may catch it. Match `OAI-SearchBot` explicitly. - The `robots.txt;` marker appears when OpenAI fetches your robots.txt. OpenAI added it so you can tell those requests apart ["especially when their logs do not include paths"](https://developers.openai.com/api/docs/bots), which is the situation on many CDN dashboards. GPTBot may add the same marker. - ChatGPT-User's link points to `https://openai.com/bot`, not to a page named after the bot. Don't build a rule on the URL in the string. ## robots.txt for "ChatGPT search yes, training no" ``` User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / Disallow: /account/ Disallow: /cart/ User-agent: * Disallow: /account/ Disallow: /cart/ Sitemap: https://example.com/sitemap.xml ``` You don't strictly need the `OAI-SearchBot` group: without one, it follows `User-agent: *`. Writing it out has one benefit. Someone who later adds `Disallow: /` to the `*` group during a staging accident or a "block all bots" panic won't take you out of ChatGPT search along with it. The cost is that the group replaces `*` for OAI-SearchBot, because [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) has a crawler obey only the most specific group that names it. Copy every private path into it, as above. The reverse policy, training allowed but no ChatGPT search, looks like this: ``` User-agent: OAI-SearchBot Disallow: / User-agent: GPTBot Allow: / ``` And a site that wants OpenAI out entirely: ``` User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User Disallow: / ``` The `ChatGPT-User` line is a request, not a guarantee, given OpenAI's own wording. If you need user-triggered fetches stopped, block the token at your firewall, preferably combined with the IP list so you don't block impostors by name and let them in by IP. ### Path-level rules Both bots honour `Allow` and `Disallow` with paths, so you can be selective. A common case is keeping a section out of training while letting all of it into search: ``` User-agent: GPTBot Allow: /blog/ Disallow: / User-agent: OAI-SearchBot Allow: / ``` The longest matching rule wins, so `/blog/` is open to GPTBot and the rest of the site is closed. OAI-SearchBot sees everything. ## How long a change takes For search, OpenAI says ["it can take ~24 hours from a site's robots.txt update for our systems to adjust"](https://developers.openai.com/api/docs/bots). OpenAI doesn't give a figure for GPTBot. Two things to do after a change: 1. Fetch your live file from outside your network: `curl -s https://example.com/robots.txt`. If Cloudflare's [managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) is on, it adds a `User-agent: GPTBot` / `Disallow: /` group to what you wrote, and that is what OpenAI reads. 2. A day later, check the logs (next section) for OAI-SearchBot hits on the `robots.txt` URL and on the paths you changed. ## Check that the traffic is really OpenAI's Anybody can send `GPTBot` in a header. OpenAI publishes IP ranges for every agent and recommends ["allowing requests from our published IP ranges"](https://developers.openai.com/api/docs/bots) for OAI-SearchBot. OpenAI doesn't document reverse DNS for these bots, so the IP files are the check. Count hits per OpenAI agent in an nginx or Apache combined log: ```bash awk -F'"' '{print $6}' /var/log/nginx/access.log \ | grep -oE 'OAI-SearchBot|GPTBot|ChatGPT-User|OAI-AdsBot' \ | sort | uniq -c | sort -rn ``` List the IPs claiming to be GPTBot: ```bash grep 'GPTBot' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20 ``` Then test those IPs against the matching JSON file. All four files use the same format, `{"creationTime": ..., "prefixes": [{"ipv4Prefix": "..."}]}`, so one script covers them: ```python # check_openai_ip.py usage: python3 check_openai_ip.py gptbot 20.171.206.10 import ipaddress, json, sys, urllib.request name, ip = sys.argv[1], ipaddress.ip_address(sys.argv[2]) url = f"https://openai.com/{name}.json" # gptbot, searchbot, chatgpt-user, adsbot data = json.load(urllib.request.urlopen(url)) nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in data["prefixes"]] print("OpenAI" if any(ip in n for n in nets) else "NOT in OpenAI's published ranges") ``` The files change. When we fetched them on 2 November 2026, `gptbot.json` had a `creationTime` of 22 September 2026 and `chatgpt-user.json` one of 27 October 2026. Don't paste the ranges into a firewall rule once and forget them. Refetch them on a schedule, the same way Perplexity [tells site owners to](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) for its own lists. A request that says GPTBot from an IP outside `gptbot.json` is someone else's scraper. It doesn't tell you anything about OpenAI's compliance, and you can block it without affecting ChatGPT. ## When your robots.txt says yes and OpenAI still can't get in robots.txt is a request. Your CDN, WAF and server decide what gets through, and each can block OpenAI while robots.txt allows it: - **Cloudflare.** Its [AI bot policies](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) sort bots into Search, Agent and Training, and a Block on Training stops GPTBot at the edge whatever your robots.txt says. Since 15 September 2026, new Cloudflare domains default to blocking Training and Agent bots on pages that display ads, and Agent covers "chat fetch bots" such as ChatGPT-User. AI Crawl Control's Overview tab [shows allowed and unsuccessful requests per crawler](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/), which is the fastest way to see whether OAI-SearchBot gets `200`s or `403`s. - **Rate limits.** A per-IP rate limit tuned for humans can throttle a crawler that comes from a few Azure ranges. - **Bot challenges.** A JavaScript challenge page returns `200` to a bot with no content. Your logs say success; OpenAI got an interstitial. You can reproduce what OpenAI gets with the token in the user agent: ```bash curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" \ -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \ https://example.com/some-page ``` This only tests user-agent rules. Rules keyed to OpenAI's IP ranges or to bot scores won't trigger from your laptop, so check the CDN's logs too. SEOFix runs the same test as part of an audit. Its `AI_CRAWLER_BLOCKED` check reads your robots.txt the way GPTBot does (its own group, or `*` without one) and flags a disallowed home page, and on full audits where the AI-crawler sample is enabled, `SLOW_FOR_AI_CRAWLERS` re-fetches up to 20 pages with GPTBot's user agent and flags pages that answer more than twice as slowly as in the normal crawl. Both are described in [AI visibility and rendering checks](https://seofix.ai/help/checks-ai-and-rendering). Note that the check looks at GPTBot, not OAI-SearchBot: a GPTBot warning is expected if you block training on purpose. ## What OpenAI doesn't document Some questions come up often and the bots page doesn't answer them, so we won't either: - **Whether OAI-SearchBot or GPTBot run JavaScript.** Not documented. Independent measurements say GPTBot doesn't; see [whether AI crawlers render JavaScript](https://seofix.ai/blog/do-ai-crawlers-render-javascript) for the evidence and a test. On a full audit, SEOFix can render a sample of up to 20 pages and flag content that only exists after JavaScript runs ([JavaScript rendering and AI crawler checks](https://seofix.ai/help/javascript-rendering)). - **Crawl rate or** **`Crawl-delay`** **support.** Not mentioned. If OpenAI's crawlers load your server, answer `429` or `503` to slow any polite crawler, or rate-limit by the published IPs. - **Whether llms.txt is read.** Not mentioned on the bots page. Our [llms.txt post](https://seofix.ai/blog/does-llms-txt-work) has the data on who reads it. ## FAQ ### If I block GPTBot, will my site disappear from ChatGPT? No. ChatGPT search depends on OAI-SearchBot. Blocking GPTBot only signals that your content ["should not be used in training generative AI foundation models"](https://developers.openai.com/api/docs/bots). ### Should I block ChatGPT-User? Only if you don't want ChatGPT users to read your pages through ChatGPT at all. Each request is a person asking about your page. OpenAI also says robots.txt may not apply to it, so a firewall rule is the only reliable block. ### Does OpenAI run other crawlers? The bots page lists four agents: OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User. Anything else claiming to be OpenAI isn't documented there, so treat it as unverified until it appears on that page and in a published IP list. --- SEOFix checks your robots.txt for GPTBot and the other AI crawlers on every audit, and its fix tasks come with the exact line to change. The first audit is free, up to 500 pages: [run it on your site](https://seofix.ai/).