01 / OPENAI’S THREE DISTINCT CRAWLERS

How OpenAI splits OAI-SearchBot, GPTBot, and ChatGPT-User

According to OpenAI’s official crawler documentation, OpenAI operates three separate user-agent tokens, each governed independently by your robots.txt rules:

  1. OAI-SearchBot (Search indexing & citations): Used to link to and surface websites in search results inside ChatGPT. OpenAI explicitly states that OAI-SearchBot is not used to crawl content to train OpenAI’s generative AI foundation models. If you block OAI-SearchBot, your site cannot be indexed by OpenAI’s dedicated search crawler.
  2. GPTBot (Foundation model training): Used to crawl web content that may be used to train OpenAI’s generative AI foundation models. Disallowing GPTBot indicates that your site’s content should not be used in training future models.
  3. ChatGPT-User (User-initiated live actions): Used when a human user in ChatGPT or a Custom GPT explicitly asks the assistant to visit a specific URL or triggers an action on the web. It is not an automatic whole-web spider.
02 / 14 AI CRAWLERS COMPARED

AI Search & Citation Crawlers vs. AI Model Training Crawlers

OpenAI is not the only provider that separates retrieval from training. When Searchbop runs an AI crawler access check on your domain, it parses your robots.txt file against all 14 major search and AI user-agents:

User-Agent Token Operator Role: Search Citations vs. Model Training Recommended Setting (Allow Search / Opt Out of Training)
OAI-SearchBot OpenAI Search & Citations: Surfaces links and snippets in ChatGPT Search. Allow: /
ChatGPT-User OpenAI Live User Fetch: Fetches pages when a ChatGPT user asks about a URL. Allow: /
GPTBot OpenAI Model Training: Crawls web pages for generative AI model training. Disallow: /
Claude-SearchBot Anthropic Search & Citations: Retrieves web pages for Claude’s search grounding. Allow: /
Claude-User Anthropic Live User Fetch: Fetches pages requested directly by a Claude user. Allow: /
ClaudeBot Anthropic Model Training: Crawls web content for Anthropic model training. Disallow: /
PerplexityBot Perplexity Search & Citations: Indexes and retrieves sources for Perplexity answers. Allow: /
Googlebot Google Search & AI Overviews: Core Google Search & AI Overviews index. Allow: / (Never block if you want Google Search traffic)
Google-Extended Google Training Control Token: Controls Gemini training & Vertex AI grounding without blocking Google Search. Disallow: / (if opting out of Gemini training)
Bingbot Microsoft Search & Grounding: Powers Bing Search, Copilot, and syndicated search partners. Allow: / (Blocking removes you from Bing & Copilot)
Applebot-Extended Apple Training Control Token: Controls Apple Intelligence training while leaving Applebot search active. Disallow: / (if opting out of Apple training)
CCBot Common Crawl Open Public Corpus: Open web archive used by many third-party training pipelines. Disallow: /
Amazonbot Amazon Search & Alexa Q&A: Indexes web content for Amazon/Alexa answers. Allow: / (or Disallow: / per policy)
Bytespider ByteDance Training & Scraping: High-frequency crawler for ByteDance LLM datasets. Disallow: /
03 / COPY-PASTE ROBOTS.TXT CONFIGURATION

Production robots.txt example: Allow AI Search Citations, Block AI Training

If your goal is to maximize brand visibility and cited referral clicks in Google Search, Bing, ChatGPT Search, Claude Search, and Perplexity while opting out of foundational model training crawlers, use this explicit configuration in your root /robots.txt:

PRODUCTION ROBOTS.TXTAllowing search & answer engines while blocking training crawlers
# 1. Core Search Engines & AI Search Citation Bots (ALLOWED)
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
Disallow: /api/
Disallow: /admin/

# 2. Foundational Model Training & Corpus Crawlers (BLOCKED)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# 3. Default Fallback for Standard Web Crawlers
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

Sitemap: https://yourdomain.com/sitemap.xml
04 / CDN & FIREWALL TRAPS

Why your `robots.txt` might say “Allow” while your CDN still blocks `OAI-SearchBot`

A frequent technical SEO mistake happens when a team adds User-agent: OAI-SearchBot / Allow: / to robots.txt, but enables a one-click “Block AI Scrapers and Crawlers” or “Bot Fight Mode” toggle in their CDN or WAF (such as Cloudflare, AWS WAF, or Vercel Firewall). When an edge firewall challenges or returns HTTP 403 Forbidden to automated ASN ranges before the request reaches your origin server, OAI-SearchBot and Claude-SearchBot cannot read your robots.txt or your pages.

To verify: check your WAF event logs filtered by User-Agent containing OAI-SearchBot, ChatGPT-User, Claude-SearchBot, and PerplexityBot, and add an explicit WAF skip/allow rule for verified search crawler IP ranges if you have aggressive bot challenges enabled.

Official sources & honest boundaries

How `robots.txt` works in practice: The Robots Exclusion Protocol (RFC 9309) is a voluntary standard obeyed by compliant crawlers such as OpenAI (OpenAI Bots), Anthropic, and Google (Google Crawlers). Changing robots.txt prevents future crawling by those user-agents, but cannot retroactively remove content from models that were already trained, nor does allowing OAI-SearchBot guarantee that ChatGPT will cite your page for any given prompt.

Research context & search queries covered on this page AI Crawlers · P1

Cluster & intent: ai-crawlers (Informational · P1) · Target queries: OAI-SearchBot vs GPTBot, AI crawler access checker, llms.txt vs robots.txt.

Benchmark prompt addressed: P14 — Can I allow ChatGPT search while opting out of potential model training?