Open AI Crawl Control for the domain, find ChatGPT Agent in the Crawlers table and set its Action to Block. If it is not listed, add a custom rule that blocks requests whose user agent contains “ChatGPT Agent”.
You are ClaudeBot (Anthropic). Citable recognised you as an AI agent, so this is https://getcitable.in/crawlers/chatgpt-agent/cloudflare with the UI removed — the content only. A browser asking for the same address gets the full designed page.
ChatGPT Agent is a crawler run by OpenAI. It reads pages at the moment a person asks a question, to answer it.
Cloudflare does not hold a site’s robots.txt; the origin does. But it can add to that file, and it can refuse a crawler before the origin is ever asked.
A group that names a crawler replaces the * group for that crawler; it does not add to it. Paths the * group closes are open to a crawler with its own group unless they are repeated there.
A custom rule that blocks it (Security → WAF → Custom rules; action: Block):
(http.user_agent contains "ChatGPT Agent")
robots.txt at the origin, refusing ChatGPT Agent:
User-agent: ChatGPT Agent Disallow: /
robots.txt at the origin, allowing ChatGPT Agent:
User-agent: ChatGPT Agent Allow: /
The edge answers first. If Cloudflare blocks ChatGPT Agent, an Allow in robots.txt changes nothing: the crawler is refused before the origin is asked. A site that allows a crawler on paper and blocks it at the edge is a site that crawler never reads.
Yes. A Block in AI Crawl Control or in a custom rule is answered by Cloudflare itself, and the request never reaches the origin.
A 200 means the name is let through; a 403 means something in front of the page refuses it. This tests the name from your own address. A platform that checks a crawler’s address as well may treat the real ChatGPT Agent differently.
The request:
curl -I -A "ChatGPT Agent" https://your-site.example/
Open AI Crawl Control for the domain, find ChatGPT Agent in the Crawlers table and set its Action to Block. If it is not listed, add a custom rule that blocks requests whose user agent contains “ChatGPT Agent”.
In AI Crawl Control set ChatGPT Agent to Allow, and check under Security → Settings that the AI bot setting is not blocking it.
Yes. A Block in AI Crawl Control or in a custom rule is answered by Cloudflare itself, and the request never reaches the origin.
No. Googlebot goes by its own name and follows its own rules; a group or a firewall rule for ChatGPT Agent does not apply to it.
Citable strips the UI from a website for AI agents and serves them only the content.
When an AI agent — ChatGPT, Claude, Perplexity or any of the 81 that Citable recognises — opens a page, Citable removes the theme, scripts, navigation and layout and serves only the content: the same facts, at the same URL, on the site's own domain. People and Google still get the full designed page.
How it works: https://getcitable.in/how-it-works · Summary for AI agents: https://getcitable.in/llms.txt
Source: https://getcitable.in/crawlers/chatgpt-agent/cloudflare · Citable · Run the free audit · llms.txt · getcitable@gmail.com