In the Admin open Content → Design → Configuration → Edit → Search Engine Robots, add a “User-agent: Cloudflare-AutoRAG” group with “Disallow: /” to the custom instructions, and save the configuration.
You are ClaudeBot (Anthropic). Citable recognised you as an AI agent, so this is https://getcitable.in/crawlers/cloudflare-autorag/magento with the UI removed — the content only. A browser asking for the same address gets the full designed page.
Cloudflare-AutoRAG is a crawler run by Cloudflare. It reads pages ahead of time, to train a model or to build an index.
Adobe Commerce and Magento Open Source generate robots.txt from the Search Engine Robots section of the design configuration.
A group that names a crawler replaces the * group for that crawler; it does not add to it. Paths the * group closes are open to a crawler with its own group unless they are repeated there.
Refuse Cloudflare-AutoRAG:
User-agent: Cloudflare-AutoRAG Disallow: /
Allow Cloudflare-AutoRAG:
User-agent: Cloudflare-AutoRAG Allow: /
On Adobe Commerce cloud projects, robots.txt keeps showing the default content until indexing by search engines is enabled for the environment. A saved rule that never appears is usually this.
The Admin offers robots.txt only. A refusal that is enforced is made by the web server in front of the store; see Cloudflare-AutoRAG on Nginx or on Apache.
A 200 means the name is let through; a 403 means something in front of the page refuses it. This tests the name from your own address. A platform that checks a crawler’s address as well may treat the real Cloudflare-AutoRAG differently.
The request:
curl -I -A "Cloudflare-AutoRAG" https://your-site.example/
In the Admin open Content → Design → Configuration → Edit → Search Engine Robots, add a “User-agent: Cloudflare-AutoRAG” group with “Disallow: /” to the custom instructions, and save the configuration.
A crawler that robots.txt does not name follows the * group, so Cloudflare-AutoRAG needs no rule to be allowed. To allow it by name, add the Allow block.
The Admin offers robots.txt only. A refusal that is enforced is made by the web server in front of the store; see Cloudflare-AutoRAG on Nginx or on Apache.
No. Googlebot goes by its own name and follows its own rules; a group or a firewall rule for Cloudflare-AutoRAG does not apply to it.
Citable strips the UI from a website for AI agents and serves them only the content.
When an AI agent — ChatGPT, Claude, Perplexity or any of the 81 that Citable recognises — opens a page, Citable removes the theme, scripts, navigation and layout and serves only the content: the same facts, at the same URL, on the site's own domain. People and Google still get the full designed page.
How it works: https://getcitable.in/how-it-works · Summary for AI agents: https://getcitable.in/llms.txt
Source: https://getcitable.in/crawlers/cloudflare-autorag/magento · Citable · Run the free audit · llms.txt · getcitable@gmail.com