Add a “User-agent: UseAI” group with “Disallow: /” to robots.txt, and to enforce it, add a RewriteCond on %{HTTP_USER_AGENT} for UseAI with a RewriteRule that ends in [F].
You are ClaudeBot (Anthropic). Citable recognised you as an AI agent, so this is https://getcitable.in/crawlers/useai/apache with the UI removed — the content only. A browser asking for the same address gets the full designed page.
UseAI is an AI crawler that identifies itself by that name. It reads pages at the moment a person asks a question, to answer it.
Apache serves robots.txt as a file from the document root. A rewrite rule, in .htaccess or the virtual host, can refuse a request by its user agent.
A group that names a crawler replaces the * group for that crawler; it does not add to it. Paths the * group closes are open to a crawler with its own group unless they are repeated there.
robots.txt, refusing UseAI:
User-agent: UseAI Disallow: /
.htaccess, refusing the request itself:
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} "\bUseAI\b" [NC]
RewriteRule ^ - [F]
</IfModule>
robots.txt, allowing UseAI:
User-agent: UseAI Allow: /
.htaccess is only read where the server’s AllowOverride permits it, and the rule needs mod_rewrite. Inside the IfModule block a missing module fails silently: the rule is simply not applied.
Yes. The [F] flag makes Apache answer 403 Forbidden itself. “UseAI” is an ordinary word, so the rule pins it to a word boundary; matched loosely it would refuse a visitor whose user agent merely contains it.
A 200 means the name is let through; a 403 means something in front of the page refuses it. This tests the name from your own address. A platform that checks a crawler’s address as well may treat the real UseAI differently.
The request:
curl -I -A "UseAI" https://your-site.example/
Add a “User-agent: UseAI” group with “Disallow: /” to robots.txt, and to enforce it, add a RewriteCond on %{HTTP_USER_AGENT} for UseAI with a RewriteRule that ends in [F].
A crawler that robots.txt does not name follows the * group, so UseAI needs no rule to be allowed. To allow it by name, add the Allow block.
Yes. The [F] flag makes Apache answer 403 Forbidden itself. “UseAI” is an ordinary word, so the rule pins it to a word boundary; matched loosely it would refuse a visitor whose user agent merely contains it.
No. Googlebot goes by its own name and follows its own rules; a group or a firewall rule for UseAI does not apply to it.
Citable strips the UI from a website for AI agents and serves them only the content.
When an AI agent — ChatGPT, Claude, Perplexity or any of the 81 that Citable recognises — opens a page, Citable removes the theme, scripts, navigation and layout and serves only the content: the same facts, at the same URL, on the site's own domain. People and Google still get the full designed page.
How it works: https://getcitable.in/how-it-works · Summary for AI agents: https://getcitable.in/llms.txt
Source: https://getcitable.in/crawlers/useai/apache · Citable · Run the free audit · llms.txt · getcitable@gmail.com