AI crawler list: 81 crawlers, who runs each, and how to allow or block it

Every AI crawler Citable recognises: who runs it, when it reads, its name in robots.txt, whether its identity can be checked, and how to allow or block it on 12 platforms.

This response

You are ClaudeBot (Anthropic). Citable recognised you as an AI agent, so this is https://getcitable.in/crawlers with the UI removed — the content only. A browser asking for the same address gets the full designed page.

The 81 crawlers

This is the list Citable's own edges match on. Googlebot and Bingbot are not on it: search crawlers are never treated as AI crawlers, and always get the real page.

• OAI-SearchBot — OpenAI. Reads while a person waits. Identity: Yes, against OpenAI’s published address lists. https://getcitable.in/crawlers/oai-searchbot
• ChatGPT-User — OpenAI. Reads while a person waits. Identity: Yes, against OpenAI’s published address lists. https://getcitable.in/crawlers/chatgpt-user
• GPTBot — OpenAI. Reads ahead of time. Identity: Yes, against OpenAI’s published address lists. https://getcitable.in/crawlers/gptbot
• Claude-SearchBot — Anthropic. Reads while a person waits. Identity: Yes, against Anthropic’s published address lists. https://getcitable.in/crawlers/claude-searchbot
• Claude-User — Anthropic. Reads while a person waits. Identity: Yes, against Anthropic’s published address lists. https://getcitable.in/crawlers/claude-user
• Claude-Web — Anthropic. Reads while a person waits. Identity: Yes, against Anthropic’s published address lists. https://getcitable.in/crawlers/claude-web
• ClaudeBot — Anthropic. Reads ahead of time. Identity: Yes, against Anthropic’s published address lists. https://getcitable.in/crawlers/claudebot
• Perplexity-User — Perplexity. Reads while a person waits. Identity: Yes, against Perplexity’s published address lists. https://getcitable.in/crawlers/perplexity-user
• PerplexityBot — Perplexity. Reads ahead of time. Identity: Yes, against Perplexity’s published address lists. https://getcitable.in/crawlers/perplexitybot
• Google-Extended — Google. A name in robots.txt, not a crawler. Identity: Nothing to check: it sends no requests. https://getcitable.in/crawlers/google-extended
• meta-externalfetcher — Meta. Reads while a person waits. Identity: Yes, by forward-confirmed reverse DNS. https://getcitable.in/crawlers/meta-externalfetcher
• meta-externalagent — Meta. Reads ahead of time. Identity: Yes, by forward-confirmed reverse DNS. https://getcitable.in/crawlers/meta-externalagent
• DuckAssistBot — DuckDuckGo. Reads while a person waits. Identity: Yes, against DuckDuckGo’s published address lists. https://getcitable.in/crawlers/duckassistbot
• CCBot — Common Crawl. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/ccbot
• Applebot-Extended — Apple. A name in robots.txt, not a crawler. Identity: Nothing to check: it sends no requests. https://getcitable.in/crawlers/applebot-extended
• Bytespider — ByteDance. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/bytespider
• Amazonbot — Amazon. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/amazonbot
• YouBot — You.com. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/youbot
• cohere-ai — Cohere. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/cohere-ai
• PetalBot — Huawei. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/petalbot
• Diffbot — Diffbot. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/diffbot
• AI2Bot — AI2. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/ai2bot
• Timpibot — Timpi. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/timpibot
• Kangaroo Bot — Kangaroo. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/kangaroo-bot
• omgili — Webz.io. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/omgili
• MistralAI-User — Mistral. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/mistralai-user
• ChatGPT Agent — OpenAI. Reads while a person waits. Identity: Yes, against OpenAI’s published address lists. https://getcitable.in/crawlers/chatgpt-agent
• Operator — OpenAI. Reads while a person waits. Identity: Yes, against OpenAI’s published address lists. https://getcitable.in/crawlers/operator
• Gemini-Deep-Research — Google. Reads while a person waits. Identity: Yes, against Google’s published address lists. https://getcitable.in/crawlers/gemini-deep-research
• Google-Agent — Google. Reads while a person waits. Identity: Yes, against Google’s published address lists. https://getcitable.in/crawlers/google-agent
• NotebookLM — Google. Reads while a person waits. Identity: Yes, against Google’s published address lists. https://getcitable.in/crawlers/notebooklm
• Kimi-User — Moonshot AI. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/kimi-user
• TongyiBot — Alibaba. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/tongyibot
• YiyanBot — Baidu. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/yiyanbot
• DoubaoBot — ByteDance. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/doubaobot
• Amzn-User — Amazon. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/amzn-user
• AmazonBuyForMe — Amazon. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/amazonbuyforme
• NovaAct — Amazon. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/novaact
• amazon-QBusiness — Amazon. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/amazon-qbusiness
• kagi-fetcher — Kagi. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/kagi-fetcher
• PhindBot — Phind. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/phindbot
• LinerBot — Liner. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/linerbot
• Manus-User — Manus. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/manus-user
• TwinAgent — Twin. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/twinagent
• ShapBot — Parallel. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/shapbot
• bigsur.ai — Big Sur AI. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/bigsur-ai
• QualifiedBot — Qualified. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/qualifiedbot
• KlaviyoAIBot — Klaviyo. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/klaviyoaibot
• Poggio-Citations — Poggio. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/poggio-citations
• iAskBot — iAsk. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/iaskbot
• Crawl4AI — Crawl4AI. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/crawl4ai
• Crawlspace — Crawlspace. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/crawlspace
• WRTNBot — WRTN. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/wrtnbot
• UseAI — UseAI. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/useai
• AIWebIndex — Lyrenth. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/aiwebindex
• Aranet-SearchBot — Aranet. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/aranet-searchbot
• KunatoCrawler — Kunato. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/kunatocrawler
• Reflectionbot — Reflection. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/reflectionbot
• GeistHaus-PageFetcher — GeistHaus. Reads while a person waits. Identity: No published list held for this vendor. https://getcitable.in/crawlers/geisthaus-pagefetcher
• MistralAI-Index — Mistral. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/mistralai-index
• Kimi-SearchBot — Moonshot AI. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/kimi-searchbot
• meta-webindexer — Meta. Reads ahead of time. Identity: Yes, by forward-confirmed reverse DNS. https://getcitable.in/crawlers/meta-webindexer
• Bravebot — Brave. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/bravebot
• ExaBot — Exa. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/exabot
• TavilyBot — Tavily. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/tavilybot
• AddSearchBot — AddSearch. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/addsearchbot
• LinkupBot — Linkup. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/linkupbot
• QueritBot — Querit. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/queritbot
• TerraCotta — Ceramic AI. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/terracotta
• HenkBot — Valyu. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/henkbot
• Channel3Bot — Channel3. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/channel3bot
• AzureAI-SearchBot — Microsoft. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/azureai-searchbot
• Amzn-SearchBot — Amazon. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/amzn-searchbot
• atlassian-bot — Atlassian. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/atlassian-bot
• Cloudflare-AutoRAG — Cloudflare. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/cloudflare-autorag
• Andibot — Andi. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/andibot
• Anomura — Direqt. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/anomura
• YandexAdditional — Yandex. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/yandexadditional
• GoogleOther — Google. Reads ahead of time. Identity: Yes, against Google’s published address lists. https://getcitable.in/crawlers/googleother
• ZanistaBot — Zanista. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/zanistabot
• Poseidon Research Crawler — Poseidon Research. Reads ahead of time. Identity: No published list held for this vendor. https://getcitable.in/crawlers/poseidon-research-crawler

By platform

• Shopify (Hosted store): Shopify generates robots.txt for every store. It is changed by adding a robots.txt.liquid template to the theme; there is no file to upload. https://getcitable.in/crawlers/on/shopify
• WordPress (Self-hosted CMS): WordPress serves a virtual robots.txt when there is no robots.txt file in the site’s root. Once a file exists the web server answers with it, and WordPress is never asked. https://getcitable.in/crawlers/on/wordpress
• Wix (Hosted site builder): Wix generates robots.txt and lets the site’s owner edit it in the Robots.txt Editor in the dashboard. https://getcitable.in/crawlers/on/wix
• Squarespace (Hosted site builder): Squarespace writes robots.txt itself and does not let it be edited. The one control is a checkbox that adds a fixed list of AI crawlers to it. https://getcitable.in/crawlers/on/squarespace
• Webflow (Hosted site builder): Webflow builds robots.txt from the rules entered in Site settings, and publishes it with the site. https://getcitable.in/crawlers/on/webflow
• BigCommerce (Hosted store): BigCommerce generates robots.txt and combines the store’s own rules with its system defaults. https://getcitable.in/crawlers/on/bigcommerce
• Adobe Commerce (Magento) (Self-hosted store): Adobe Commerce and Magento Open Source generate robots.txt from the Search Engine Robots section of the design configuration. https://getcitable.in/crawlers/on/magento
• Next.js (Framework): In the App Router, robots.txt is either a static file or the output of app/robots.js, a function that returns the rules. https://getcitable.in/crawlers/on/nextjs
• Cloudflare (Edge in front of the site): Cloudflare does not hold a site’s robots.txt; the origin does. But it can add to that file, and it can refuse a crawler before the origin is ever asked. https://getcitable.in/crawlers/on/cloudflare
• Vercel (Hosting with a firewall in front): On Vercel robots.txt is part of the project: a file in public/, or whatever the framework generates. In front of the project sits the Firewall, which can refuse a crawler before the project is asked. https://getcitable.in/crawlers/on/vercel
• Nginx (Web server): Nginx serves robots.txt as a file from the site’s root, like any other file. The server itself can also refuse a request by its user agent. https://getcitable.in/crawlers/on/nginx
• Apache (Web server): Apache serves robots.txt as a file from the document root. A rewrite rule, in .htaccess or the virtual host, can refuse a request by its user agent. https://getcitable.in/crawlers/on/apache

Two things a robots.txt rule does not do

A group that names a crawler replaces the * group for that crawler; it does not add to it. Paths the * group closes are open to a crawler with its own group unless they are repeated there.

robots.txt is a request. A crawler that honours it stays out; nothing in the file stops one that does not. What is enforced is what the server or the edge in front of it refuses.

About Citable

Citable strips the UI from a website for AI agents and serves them only the content.

When an AI agent — ChatGPT, Claude, Perplexity or any of the 81 that Citable recognises — opens a page, Citable removes the theme, scripts, navigation and layout and serves only the content: the same facts, at the same URL, on the site's own domain. People and Google still get the full designed page.

How it works: https://getcitable.in/how-it-works · Summary for AI agents: https://getcitable.in/llms.txt

Source: https://getcitable.in/crawlers · Citable · Run the free audit · llms.txt · getcitable@gmail.com