GPTBot, OAI-SearchBot, Google-Extended, and llms.txt: What Each Controls

For years, a publisher could treat crawler policy as one mostly binary question: should Googlebot index this page? That model no longer fits. The same public documentation may be requested for traditional search, an AI answer engine, future model training, grounded generation, or a one-off fetch initiated by a human. Those uses can have different business value and different user-agent controls.

The dangerous response is a blanket User-agent: * block. It may protect nothing from a non-compliant scraper while simultaneously removing the legitimate crawlers that help customers find you. The opposite extreme—allowing everything because the site is public—may not match your intellectual-property policy. A defensible 2026 strategy is granular: decide by purpose, write exact rules, and verify that your CDN and origin enforce the same decision.

The publisher's dilemma: discovery and reuse are not one switch

Imagine a developer-platform company with teams in New York, Toronto, and London. Its public documentation contains the clearest explanation of a proprietary workflow. Search visibility brings trial users, support deflection, and qualified referrals. At the same time, the company may not want that content collected for a future model-training pipeline.

A blanket allow policy ignores the second goal; a blanket block ignores the first. The right conversation is not “Do we allow AI?” It is “Which uses do we permit, on which paths, and what outcome do we expect?” Write the answer down before touching robots.txt.

The AI crawler directory: know which token controls which use

GPTBot: OpenAI model improvement and training

OpenAI describes GPTBot as a crawler for content that may be used to make its generative AI foundation models more useful and safe. A site can express that its content should not be used for that training purpose by disallowing GPTBot in robots.txt.

Do not describe this as a switch for a specific product release such as “GPT-5 training.” Model names, versions, and data pipelines change. The durable statement is the one the crawler owner documents: GPTBot is the control for OpenAI's foundation-model training use. It is independent from OAI-SearchBot.

User-agent: GPTBot
Disallow: /

OAI-SearchBot: ChatGPT search discovery and surfacing

OAI-SearchBot is OpenAI's automatic crawler for search. OpenAI says sites opted out of it will not be shown in ChatGPT search answers, although a URL may still appear as a navigational link. To make content eligible for summaries, snippets, and links, allow the bot in robots.txt and make sure requests from OpenAI's published IP ranges can pass your CDN, WAF, and origin.

“Eligible” is the critical word. An allow rule cannot force a crawl, inclusion, answer citation, ranking position, or referral visit. Content quality, relevance, technical delivery, and the search system's own selection still matter. OpenAI also notes that a robots.txt update can take roughly 24 hours to be reflected by its search systems.

User-agent: OAI-SearchBot
Allow: /
Disallow: /api/
Disallow: /account/
Disallow: /private-preview/

ChatGPT-User: a user-triggered request, not an automatic crawl

A person can ask ChatGPT or a Custom GPT to open a URL. Those requests may use ChatGPT-User. OpenAI says this agent is not used for automatic web crawling or to determine Search inclusion, and because the request is initiated by a user, robots.txt rules may not apply.

That distinction matters for security. If a URL must never be fetched without permission, protect it with authentication. If you want to control automated resource consumption, add verified-bot policies, behavioral rate limits, and WAF rules. A robots declaration alone cannot reliably enforce either requirement.

Google-Extended: a product token, not a visible crawler

Google-Extended is commonly called an AI crawler, but Google's current documentation is more precise: it has no separate HTTP user-agent string. Google crawls with existing user agents and interpretsGoogle-Extended as a standalone robots.txt product token.

The token controls whether crawled content may be used to train future generations of Gemini models powering Gemini Apps and the Vertex AI API for Gemini, and for grounding with Google Search in Gemini Apps and Vertex AI. Google explicitly says the setting does not affect a site's inclusion in Google Search and is not a Search ranking signal.

User-agent: Google-Extended
Disallow: /

A practical robots.txt configuration for search without training

The following starting point fits a public technology company that wants Google Search and ChatGPT search discovery while expressing an opt-out preference for the documented OpenAI and Google training uses. Replace the example paths and sitemap URL with your own. Do not paste it over an existing file without reconciling its current rules.

# OpenAI search: eligible for ChatGPT search discovery
User-agent: OAI-SearchBot
Allow: /
Disallow: /api/
Disallow: /account/
Disallow: /private-preview/

# OpenAI training preference
User-agent: GPTBot
Disallow: /

# Google Gemini training and grounding preference
User-agent: Google-Extended
Disallow: /

# Standard Google Search crawling
User-agent: Googlebot
Allow: /
Disallow: /api/
Disallow: /account/
Disallow: /private-preview/

Sitemap: https://www.example.com/sitemap.xml

Explicit Allow: / lines make the intent readable to humans, even though crawling is generally allowed when no matching disallow rule exists. Separate groups with blank lines, preserve the exact documented token spelling, and publish the file at the origin root:https://www.example.com/robots.txt. Each hostname or subdomain needs the policy served from its own root.

Five deployment mistakes to catch before production

1. Blocking every bot with a wildcard. AUser-agent: * plus Disallow: / can remove legitimate search crawling as collateral damage.
2. Assuming an allow rule bypasses the WAF. A crawler can be permitted in robots.txt and still receive a 403, JavaScript challenge, geo-block, login screen, or rate-limit response.
3. Using robots.txt to prevent indexing. A blocked URL may still be known through links and shown without a crawled snippet. Use an appropriate noindex directive when removal from search results is the actual goal, and allow the crawler to read it.
4. Publishing private paths in a public policy.Disallowed paths advertise their names. Authentication, authorization, and signed access remain mandatory.
5. Testing only the file contents. A syntactically valid robots.txt file can be cached incorrectly, served only on one hostname, redirected unexpectedly, or contradicted by an edge rule.

How to verify crawler policy through the full delivery stack

Validation must cover both interpretation and delivery. First, fetch the production robots.txt file without a browser cache. Then test a public page with representative user-agent strings. Finally, inspect logs to see the status codes real requests received.

# Fetch the production policy and response headers
curl -i https://www.example.com/robots.txt

# Test what the origin/CDN serves to OpenAI search
curl -I   -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)"   https://www.example.com/docs/getting-started

# Test an ordinary Googlebot-shaped request
curl -I   -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"   https://www.example.com/docs/getting-started

A copied user-agent string is useful for finding obvious CDN behavior, but it does not prove crawler identity: anyone can spoof a header. OpenAI publishes IP-range files for its crawlers, while Google documents crawler verification through published ranges and reverse-DNS checks. Use those mechanisms—or your CDN's verified-bot classification—before granting a security exception.

llms.txt: Markdown for discovery, not crawler permission

The /llms.txt proposal suggests publishing a compact, human-readable Markdown file that introduces a project and links to its most useful resources. The goal is to help supporting LLM-based tools find high-signal documentation at inference time without first parsing a navigation-heavy website.

It is an emerging community convention, not a formal IETF or W3C web standard. No universal crawler behavior is attached to it, and no major AI product guarantees that publishing the file improves retrieval, ranking, or citations. That makes it a low-risk documentation experiment when maintained carefully—not a replacement for technical SEO.

The proposal-aligned structure

The core proposal requires only an H1 project or site name. A concise blockquote summary, optional explanatory text, and H2 sections containing Markdown link lists add useful context. The specially namedOptional section is for secondary resources that can be skipped when a smaller context is preferred.

# Example Developer Platform

> APIs and SDKs for validating structured data in developer workflows.

Use these resources for current product behavior. The API reference is
authoritative for endpoints, schemas, and error responses.

## Start here

- [Quickstart](https://www.example.com/docs/quickstart.md): Create a project and send the first request.
- [Authentication](https://www.example.com/docs/authentication.md): API key scopes, storage, and rotation.

## Reference

- [REST API](https://www.example.com/docs/api.md): Endpoints, parameters, examples, and errors.
- [Changelog](https://www.example.com/changelog): Dated product and API changes.

## Optional

- [Full documentation](https://www.example.com/llms-full.txt): Expanded one-file context for supported agents.

Some implementations publish /llms-full.txt as a larger, one-file companion. Treat that filename as an optional ecosystem convention rather than a required part of the core proposal. For a large documentation site, the full file can become expensive and stale; a concise index with accurate descriptions is usually the better first artifact.

What belongs in llms.txt—and what does not

• Link to canonical, current pages: quickstarts, API references, authentication, pricing, security, policies, changelogs, and support.
• Use specific descriptions that explain why an agent should open each resource. “API reference with endpoint schemas and error examples” is better than “Docs.”
• Keep the index curated. A dump of every URL duplicates a sitemap and defeats the high-signal purpose.
• Never place secrets, unpublished roadmaps, personal data, hidden instructions, or text that conflicts with the canonical page in it.
• Update the file when URLs, product names, authentication flows, or supported versions change. Add it to link-checking and release review.

A production audit workflow

1. Inventory public content. Separate marketing pages, docs, support, community content, paid material, account pages, APIs, and private previews. Anything truly private must move behind access control before crawler policy is discussed.
2. Record decisions by purpose. Decide independently for traditional search, AI search, model training, grounding, archival collection, and user-triggered retrieval. Get legal or content-owner review where policy requires it.
3. Generate the smallest correct robots change. Add exact groups for GPTBot, OAI-SearchBot, and Google-Extended without erasing existing rules for Googlebot, images, ads, or application paths.
4. Test representative URLs per agent. Confirm homepage, documentation, assets, API, account, and private-preview behavior. Validate both rule matching and the response returned by the CDN.
5. Verify real crawler identity. Use official IP data, reverse DNS where documented, or a trusted verified-bot feature. Never whitelist an unverified request solely because its header says GPTBot or Googlebot.
6. Publish llms.txt only if someone owns it. Keep it concise, link to canonical sources, and include it in documentation tests. Omit it if it will become an abandoned duplicate.
7. Monitor outcomes. Watch status codes, request volume, bytes transferred, response time, path distribution, crawl changes, search referrals, and dead links. Recheck vendor documentation during quarterly technical-SEO reviews.

Conclusion: crawler policy is now a content API

Modern crawler management is not a war between “allow AI” and “block AI.” It is a routing policy for public content. GPTBot, OAI-SearchBot, Google-Extended, Googlebot, and ChatGPT-User represent different purposes and, in some cases, different enforcement limits. Treating them as interchangeable creates both discoverability loss and false confidence.

Start with explicit business decisions, deploy exact robots groups, and verify the edge rather than trusting a file in source control. Then add llms.txt only as a maintained, experimental map for tools that choose to read it. If the real requirement is removing a page from results rather than controlling crawling, continue with our guide to robots.txt, meta robots, and X-Robots-Tag—the distinction prevents some of the web's most expensive indexing mistakes.