OAI-SearchBot, GPTBot, and ChatGPT-User: What to Allow in robots.txt

OpenAI crawler access is no longer a single yes-or-no decision. The important question is not "Should we block OpenAI?" The better question is: "Which OpenAI agent are we controlling, and what outcome do we want?"

OpenAI documents separate agents for separate jobs:

• OAI-SearchBot: used for surfacing websites in ChatGPT search features.
• GPTBot: used for crawling content that may be used to train OpenAI generative AI foundation models.
• ChatGPT-User: used for certain user-triggered actions in ChatGPT and Custom GPTs.

That separation matters for every company doing GEO, LLM visibility, AI search optimization, or AI traffic measurement. If you block the wrong agent, you can reduce your chance of appearing in ChatGPT search. If you allow everything by accident, you may be allowing training access that your legal or content team intended to restrict.

The short answer

If you want visibility in ChatGPT search but do not want your content used for OpenAI model training, the common policy is:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

This allows OpenAI's search crawler while disallowing GPTBot training access. ChatGPT-User is different because OpenAI describes it as user-triggered rather than automatic web crawling, and notes that robots.txt rules may not apply to those user actions.

For many public marketing sites, docs sites, SaaS landing pages, ecommerce pages, and comparison pages, allowing OAI-SearchBot is the most important rule for ChatGPT search discoverability.

OpenAI crawler roles at a glance

Agent	Main purpose	robots.txt control	Recommended default for growth sites
OAI-SearchBot	ChatGPT search visibility and search result surfacing	Yes	Allow on public pages you want discoverable
GPTBot	Training data collection for generative AI foundation models	Yes	Allow or disallow based on your training policy
ChatGPT-User	User-triggered ChatGPT and Custom GPT actions	May not apply in the same way	Allow public pages unless you have a specific access-control reason

What is OAI-SearchBot?

OAI-SearchBot is OpenAI's search crawler. OpenAI says it is used to surface websites in ChatGPT search features. Sites opted out of OAI-SearchBot may not be shown in ChatGPT search answers, though they may still appear as navigational links in some cases.

For SEO and GEO teams, this is the agent that maps most directly to discoverability. If you want your public pages to be found, cited, linked, or surfaced by ChatGPT search, do not accidentally block OAI-SearchBot.

Recommended OAI-SearchBot rule

User-agent: OAI-SearchBot
Allow: /

You can also allow only specific public sections:

User-agent: OAI-SearchBot
Allow: /blog/
Allow: /docs/
Allow: /pricing
Disallow: /app/
Disallow: /dashboard/
Disallow: /account/

This is useful when you want ChatGPT search visibility for public educational and commercial pages, but not for authenticated product surfaces.

What is GPTBot?

GPTBot is different. OpenAI says GPTBot is used to crawl content that may be used in training generative AI foundation models. Disallowing GPTBot indicates that your site's content should not be used for that training purpose.

This means GPTBot is a training-policy decision, not a search-visibility decision. A site can allow OAI-SearchBot while blocking GPTBot.

Recommended GPTBot rule when you want training opt-out

User-agent: GPTBot
Disallow: /

For companies with public docs, blog posts, or marketing content, this distinction is useful. You can pursue ChatGPT search visibility without automatically making the same decision about model training access.

What is ChatGPT-User?

ChatGPT-User is used for certain actions initiated by users in ChatGPT and Custom GPTs. OpenAI says that when users ask ChatGPT or a Custom GPT a question, it may visit a web page with the ChatGPT-User agent. Users may also interact with external applications through GPT Actions.

The important difference: ChatGPT-User is not described as automatic web crawling. OpenAI also says it is not used to determine whether content may appear in Search. For Search opt-outs and automatic crawl control, OAI-SearchBot is the relevant agent.

Recommended ChatGPT-User rule for public sites

User-agent: ChatGPT-User
Allow: /

If a page should not be accessed by users or agents, do not rely only on robots.txt. Use authentication, authorization, noindex where appropriate, and normal security controls.

Three common robots.txt policies

Policy 1: Maximize ChatGPT search visibility

Use this when your priority is discoverability and you are comfortable allowing search and training crawls on public pages.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Policy 2: Allow ChatGPT search, block model training

Use this when your growth team wants ChatGPT search visibility, but your content, legal, or executive team wants to opt out of training access.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

Policy 3: Allow only public content sections

Use this when you want AI search access for public pages but want to keep application routes, dashboards, accounts, and private surfaces out of crawl paths.

User-agent: OAI-SearchBot
Allow: /blog/
Allow: /docs/
Allow: /use-cases/
Allow: /pricing
Disallow: /dashboard/
Disallow: /account/
Disallow: /api/

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /blog/
Allow: /docs/
Allow: /use-cases/
Allow: /pricing

This is often the most practical B2B SaaS policy: public knowledge stays discoverable; private product areas stay controlled.

Robots.txt mistakes that hurt AI visibility

• Blocking all OpenAI agents as one group: this can block search visibility when you only meant to block training access.
• Blocking OAI-SearchBot on marketing pages: this reduces your chance of surfacing in ChatGPT search features.
• Assuming GPTBot controls ChatGPT search: GPTBot is a training crawler, while OAI-SearchBot is the search crawler.
• Relying on robots.txt for private data: use authentication and access control for anything truly private.
• Forgetting CDN and firewall rules: allowing a bot in robots.txt is not enough if your edge layer blocks the request.

How this connects to LLM visibility audits

For AI visibility, crawlability is the first layer. If an assistant search crawler cannot access the page, the rest of your GEO work becomes weaker. Your structured data, FAQ sections, comparison tables, and answer-first content matter only if the relevant crawler can fetch and parse them.

Aparok helps teams audit this layer as part of LLM readiness: crawlability, extractable content, page structure, trust signals, and whether public pages are likely to be usable by AI discovery systems. Robots.txt is one of the first places to check because it can silently block the exact pages you want surfaced.

OpenAI crawler audit checklist

1. Open your robots.txt: check /robots.txt on your production domain.
2. Find OpenAI-specific rules: look for OAI-SearchBot, GPTBot, and ChatGPT-User.
3. Confirm search policy: public pages you want in ChatGPT search should not block OAI-SearchBot.
4. Confirm training policy: decide separately whether GPTBot should be allowed or disallowed.
5. Check app/private paths: dashboards, accounts, admin paths, and APIs should usually be disallowed or protected.
6. Check noindex tags: a page can be crawlable but still excluded from indexing if it has noindex.
7. Check firewall/CDN rules: verify the bot is not blocked before it reaches your app.
8. Run an Aparok audit: use crawl-based LLM readiness checks to see whether your public pages are extractable and aligned with AI discovery.

Recommended setup for most SaaS and content sites

For most public-facing SaaS sites, the practical default is:

• Allow OAI-SearchBot on marketing, blog, docs, pricing, comparison, and use-case pages.
• Disallow GPTBot if your company wants to opt out of training access.
• Allow ChatGPT-User on public pages users may ask ChatGPT to inspect.
• Disallow private app routes and protect them with authentication.
• Keep sitemap.xml, canonical tags, schema, and visible FAQ content aligned with the pages you want surfaced.

That setup preserves ChatGPT search visibility while keeping the training question separate.

FAQ

Should I allow OAI-SearchBot?

If you want your public pages to appear in ChatGPT search features, yes. OpenAI recommends allowing OAI-SearchBot for search visibility.

Can I block GPTBot but still appear in ChatGPT search?

Yes. OpenAI documents OAI-SearchBot and GPTBot as separate controls. You can allow OAI-SearchBot for search and disallow GPTBot for training opt-out.

Does GPTBot control ChatGPT search visibility?

No. GPTBot is for content that may be used in training generative AI foundation models. OAI-SearchBot is the crawler tied to ChatGPT search surfacing.

Should I block ChatGPT-User?

Usually not on public pages. ChatGPT-User is used for certain user-triggered actions, not automatic search crawling. If a page is private, use authentication and access control rather than relying only on robots.txt.

How long does a robots.txt update take for OpenAI search?

OpenAI says search-result systems can take about 24 hours from a robots.txt update to adjust.

Can Aparok help audit this?

Yes. Aparok's AI discovery workflow includes crawl-based readiness audits, so teams can find crawlability, structure, and extractability issues before they spend time writing more content.

Bottom line

Treat OpenAI crawler access as three separate decisions. Allow OAI-SearchBot if you want ChatGPT search visibility. Decide separately whether GPTBot should access your content for training. Treat ChatGPT-User as user-triggered access and protect private pages with real access controls.

Use Aparok to audit whether your site is crawlable, extractable, and ready for AI discovery across ChatGPT, Perplexity, Gemini, and other assistant surfaces.