AI crawlers on Next.js: allow or block GPTBot, ClaudeBot, PerplexityBot and 78 more

In the App Router, robots.txt is either a static file or the output of app/robots.js, a function that returns the rules. The steps on Next.js, what trips people up there, and a page for each of 81 AI crawlers.

This response

You are ClaudeBot (Anthropic). Citable recognised you as an AI agent, so this is https://getcitable.in/crawlers/on/nextjs with the UI removed — the content only. A browser asking for the same address gets the full designed page.

Where Next.js keeps the rule

In the App Router, robots.txt is either a static file or the output of app/robots.js, a function that returns the rules.

The steps below are shown for GPTBot. Each crawler has its own page, with its own names in the rule.

Step by step

What to paste

A group that names a crawler replaces the * group for that crawler; it does not add to it. Paths the * group closes are open to a crawler with its own group unless they are repeated there.

app/robots.js, refusing GPTBot:

export default function robots() {
  return {
    rules: [
      { userAgent: "GPTBot", disallow: "/" },
      { userAgent: "*", allow: "/" },
    ],
  };
}

The same as a static robots.txt:

User-agent: GPTBot
Disallow: /

proxy.js, refusing the request itself:

import { NextResponse } from "next/server";

const REFUSED = /GPTBot/i;

// Next.js 16. Before it, the file is middleware.js and the function is middleware.
export function proxy(request) {
  if (REFUSED.test(request.headers.get("user-agent") || "")) {
    return new NextResponse("Forbidden", { status: 403 });
  }
  return NextResponse.next();
}

What trips people up

app/robots.js is run at build time and its output is cached. Editing it changes nothing on the live site until the next deploy. If the site is on Vercel, the Firewall sits in front of all of it.

Asked, or actually refused?

Yes: the proxy runs before the page is rendered, so a 403 from it is a refusal and not a request.

Check it yourself

A 200 means the name is let through; a 403 means something in front of the page refuses it. This tests the name from your own address. A platform that checks a crawler’s address as well may treat the real GPTBot differently.

The request:

curl -I -A "GPTBot" https://your-site.example/

Every AI crawler, on Next.js

Checked against Next.js's own documentation on 1 October 2026

About Citable

Citable strips the UI from a website for AI agents and serves them only the content.

When an AI agent — ChatGPT, Claude, Perplexity or any of the 81 that Citable recognises — opens a page, Citable removes the theme, scripts, navigation and layout and serves only the content: the same facts, at the same URL, on the site's own domain. People and Google still get the full designed page.

How it works: https://getcitable.in/how-it-works · Summary for AI agents: https://getcitable.in/llms.txt

Source: https://getcitable.in/crawlers/on/nextjs · Citable · Run the free audit · llms.txt · getcitable@gmail.com