Return a rule with userAgent “Poseidon Research Crawler” and disallow “/” from app/robots.js, and for a refusal that is enforced, return a 403 from proxy.js (middleware.js before Next.js 16) when the user agent matches. Deploy for it to take effect.
You are ClaudeBot (Anthropic). Citable recognised you as an AI agent, so this is https://getcitable.in/crawlers/poseidon-research-crawler/nextjs with the UI removed — the content only. A browser asking for the same address gets the full designed page.
Poseidon Research Crawler is a crawler run by Poseidon Research. It reads pages ahead of time, to train a model or to build an index.
In the App Router, robots.txt is either a static file or the output of app/robots.js, a function that returns the rules.
A group that names a crawler replaces the * group for that crawler; it does not add to it. Paths the * group closes are open to a crawler with its own group unless they are repeated there.
app/robots.js, refusing Poseidon Research Crawler:
export default function robots() {
return {
rules: [
{ userAgent: "Poseidon Research Crawler", disallow: "/" },
{ userAgent: "*", allow: "/" },
],
};
}
The same as a static robots.txt:
User-agent: Poseidon Research Crawler Disallow: /
proxy.js, refusing the request itself:
import { NextResponse } from "next/server";
const REFUSED = /Poseidon Research Crawler/i;
// Next.js 16. Before it, the file is middleware.js and the function is middleware.
export function proxy(request) {
if (REFUSED.test(request.headers.get("user-agent") || "")) {
return new NextResponse("Forbidden", { status: 403 });
}
return NextResponse.next();
}
app/robots.js is run at build time and its output is cached. Editing it changes nothing on the live site until the next deploy. If the site is on Vercel, the Firewall sits in front of all of it.
Yes: the proxy runs before the page is rendered, so a 403 from it is a refusal and not a request.
A 200 means the name is let through; a 403 means something in front of the page refuses it. This tests the name from your own address. A platform that checks a crawler’s address as well may treat the real Poseidon Research Crawler differently.
The request:
curl -I -A "Poseidon Research Crawler" https://your-site.example/
Return a rule with userAgent “Poseidon Research Crawler” and disallow “/” from app/robots.js, and for a refusal that is enforced, return a 403 from proxy.js (middleware.js before Next.js 16) when the user agent matches. Deploy for it to take effect.
A crawler that robots.txt does not name follows the * group, so Poseidon Research Crawler needs no rule to be allowed. To allow it by name, add the Allow block.
Yes: the proxy runs before the page is rendered, so a 403 from it is a refusal and not a request.
No. Googlebot goes by its own name and follows its own rules; a group or a firewall rule for Poseidon Research Crawler does not apply to it.
Citable strips the UI from a website for AI agents and serves them only the content.
When an AI agent — ChatGPT, Claude, Perplexity or any of the 81 that Citable recognises — opens a page, Citable removes the theme, scripts, navigation and layout and serves only the content: the same facts, at the same URL, on the site's own domain. People and Google still get the full designed page.
How it works: https://getcitable.in/how-it-works · Summary for AI agents: https://getcitable.in/llms.txt
Source: https://getcitable.in/crawlers/poseidon-research-crawler/nextjs · Citable · Run the free audit · llms.txt · getcitable@gmail.com