In the project’s Firewall → Rules, add a custom rule on the User Agent parameter with the value OAI-SearchBot and the action Deny, then Review Changes and Publish.
You are ClaudeBot (Anthropic). Citable recognised you as an AI agent, so this is https://getcitable.in/crawlers/oai-searchbot/vercel with the UI removed — the content only. A browser asking for the same address gets the full designed page.
OAI-SearchBot is a crawler run by OpenAI. It reads pages at the moment a person asks a question, to answer it.
On Vercel robots.txt is part of the project: a file in public/, or whatever the framework generates. In front of the project sits the Firewall, which can refuse a crawler before the project is asked.
A group that names a crawler replaces the * group for that crawler; it does not add to it. Paths the * group closes are open to a crawler with its own group unless they are repeated there.
Refuse OAI-SearchBot:
User-agent: OAI-SearchBot Disallow: /
Allow OAI-SearchBot:
User-agent: OAI-SearchBot Allow: /
The AI Bots Ruleset is one switch for every AI bot Vercel recognises, and Vercel keeps that list. On Deny it refuses all of them, and whether OAI-SearchBot is on that list is Vercel’s decision; a custom rule is the only control that names one crawler. It is off by default, shown in the dashboard as Allow.
Yes. A Deny in the Firewall is answered by Vercel, before the project’s code runs.
A 200 means the name is let through; a 403 means something in front of the page refuses it. This tests the name from your own address. A platform that checks a crawler’s address as well may treat the real OAI-SearchBot differently.
The request:
curl -I -A "OAI-SearchBot" https://your-site.example/
In the project’s Firewall → Rules, add a custom rule on the User Agent parameter with the value OAI-SearchBot and the action Deny, then Review Changes and Publish.
Leave the AI Bots Ruleset on Allow, which is its default. If it is on Deny, add a custom rule for OAI-SearchBot’s User Agent with the action Bypass.
Yes. A Deny in the Firewall is answered by Vercel, before the project’s code runs.
No. Googlebot goes by its own name and follows its own rules; a group or a firewall rule for OAI-SearchBot does not apply to it.
Citable strips the UI from a website for AI agents and serves them only the content.
When an AI agent — ChatGPT, Claude, Perplexity or any of the 81 that Citable recognises — opens a page, Citable removes the theme, scripts, navigation and layout and serves only the content: the same facts, at the same URL, on the site's own domain. People and Google still get the full designed page.
How it works: https://getcitable.in/how-it-works · Summary for AI agents: https://getcitable.in/llms.txt
Source: https://getcitable.in/crawlers/oai-searchbot/vercel · Citable · Run the free audit · llms.txt · getcitable@gmail.com