AI search fix
OAI-SearchBot: what it is and how to allow it
OAI-SearchBot is an OpenAI user-agent used when OpenAI fetches public pages for search-oriented product flows. It sits apart from GPTBot, which handles broader ChatGPT retrieval crawling, and ChatGPT-User, which often reflects user-initiated browsing. Expect OAI-SearchBot in origin logs, CDN bot panels and Microsoft Clarity Bot Activity. To permit it, add a dedicated Allow block in robots.txt and confirm your WAF does not challenge that string. Blocking OAI-SearchBot whilst permitting GPTBot is a valid choice — document each user-agent on its own line. Permitting OAI-SearchBot does not guarantee ChatGPT search features will cite your site.
Audit OpenAI access by exporting robots.txt and filtering logs for GPTBot, OAI-SearchBot and ChatGPT-User together. A catch-all Disallow under User-agent: * can override narrower allows if your CMS merges rules incorrectly.
| User-agent | Typical role |
|---|---|
| GPTBot | Broad ChatGPT-class retrieval and indexing of public pages. |
| OAI-SearchBot | Search-oriented fetches — controlled via its own robots.txt section. |
| ChatGPT-User | Often linked to user-driven browsing — policy separately. |
| GEO prerequisite | Reachable HTML, honest copy and schema.org JSON-LD where appropriate. |
You'll get an HTML report listing GPTBot, OAI-SearchBot and related OpenAI strings in robots.txt.
Run the diagnosticRelated questions
- GPTBot user-agent: what it is and how to allow itCompare sibling OpenAI crawler policy before merging rules.
- ChatGPT cannot crawl my website — how to fix itFull crawl troubleshooting when OpenAI bots never reach HTML.
- robots.txt for AI crawlers — how to write itSection templates for OpenAI and other AI user-agents.
Updated
