AI crawlers on Squarespace: allow or block GPTBot, ClaudeBot, PerplexityBot and 78 more

Squarespace writes robots.txt itself and does not let it be edited. The one control is a checkbox that adds a fixed list of AI crawlers to it. The steps on Squarespace, what trips people up there, and a page for each of 81 AI crawlers.

This response

You are ClaudeBot (Anthropic). Citable recognised you as an AI agent, so this is https://getcitable.in/crawlers/on/squarespace with the UI removed — the content only. A browser asking for the same address gets the full designed page.

Where Squarespace keeps the rule

Squarespace writes robots.txt itself and does not let it be edited. The one control is a checkbox that adds a fixed list of AI crawlers to it.

The steps below are shown for GPTBot. Each crawler has its own page, with its own names in the rule.

Step by step

• 1. Open the setting. Open the Settings panel and click Crawlers.
• 2. To refuse GPTBot, tick the box. Tick “Block known artificial intelligence crawlers”. Squarespace then adds GPTBot to robots.txt, along with every other name on its list.
• 3. To allow GPTBot, leave it unticked. The box is unticked by default, and unticked it adds nothing to robots.txt.
• 4. Read it back. Open /robots.txt on the site and look for the name.

What trips people up

The box is all or nothing. Ticking it to refuse GPTBot also refuses the other 25 names on Squarespace’s list: AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot.

Asked, or actually refused?

Squarespace has no setting that refuses a request by its user agent. robots.txt is a request. A crawler that honours it stays out; nothing in the file stops one that does not. What is enforced is what the server or the edge in front of it refuses.

Check it yourself

A 200 means the name is let through; a 403 means something in front of the page refuses it. This tests the name from your own address. A platform that checks a crawler’s address as well may treat the real GPTBot differently.

The request:

curl -I -A "GPTBot" https://your-site.example/

Every AI crawler, on Squarespace

• OAI-SearchBot (OpenAI) — Not on Squarespace’s list: https://getcitable.in/crawlers/oai-searchbot/squarespace
• ChatGPT-User (OpenAI) — Not on Squarespace’s list: https://getcitable.in/crawlers/chatgpt-user/squarespace
• GPTBot (OpenAI) — Covered by the checkbox: https://getcitable.in/crawlers/gptbot/squarespace
• Claude-SearchBot (Anthropic) — Not on Squarespace’s list: https://getcitable.in/crawlers/claude-searchbot/squarespace
• Claude-User (Anthropic) — Not on Squarespace’s list: https://getcitable.in/crawlers/claude-user/squarespace
• Claude-Web (Anthropic) — Not on Squarespace’s list: https://getcitable.in/crawlers/claude-web/squarespace
• ClaudeBot (Anthropic) — Covered by the checkbox: https://getcitable.in/crawlers/claudebot/squarespace
• Perplexity-User (Perplexity) — Not on Squarespace’s list: https://getcitable.in/crawlers/perplexity-user/squarespace
• PerplexityBot (Perplexity) — Not on Squarespace’s list: https://getcitable.in/crawlers/perplexitybot/squarespace
• Google-Extended (Google) — Covered by the checkbox: https://getcitable.in/crawlers/google-extended/squarespace
• meta-externalfetcher (Meta) — Not on Squarespace’s list: https://getcitable.in/crawlers/meta-externalfetcher/squarespace
• meta-externalagent (Meta) — Covered by the checkbox: https://getcitable.in/crawlers/meta-externalagent/squarespace
• DuckAssistBot (DuckDuckGo) — Covered by the checkbox: https://getcitable.in/crawlers/duckassistbot/squarespace
• CCBot (Common Crawl) — Covered by the checkbox: https://getcitable.in/crawlers/ccbot/squarespace
• Applebot-Extended (Apple) — Covered by the checkbox: https://getcitable.in/crawlers/applebot-extended/squarespace
• Bytespider (ByteDance) — Covered by the checkbox: https://getcitable.in/crawlers/bytespider/squarespace
• Amazonbot (Amazon) — Covered by the checkbox: https://getcitable.in/crawlers/amazonbot/squarespace
• YouBot (You.com) — Covered by the checkbox: https://getcitable.in/crawlers/youbot/squarespace
• cohere-ai (Cohere) — Covered by the checkbox: https://getcitable.in/crawlers/cohere-ai/squarespace
• PetalBot (Huawei) — Not on Squarespace’s list: https://getcitable.in/crawlers/petalbot/squarespace
• Diffbot (Diffbot) — Not on Squarespace’s list: https://getcitable.in/crawlers/diffbot/squarespace
• AI2Bot (AI2) — Covered by the checkbox: https://getcitable.in/crawlers/ai2bot/squarespace
• Timpibot (Timpi) — Not on Squarespace’s list: https://getcitable.in/crawlers/timpibot/squarespace
• Kangaroo Bot (Kangaroo) — Not on Squarespace’s list: https://getcitable.in/crawlers/kangaroo-bot/squarespace
• omgili (Webz.io) — Covered by the checkbox: https://getcitable.in/crawlers/omgili/squarespace
• MistralAI-User (Mistral) — Not on Squarespace’s list: https://getcitable.in/crawlers/mistralai-user/squarespace
• ChatGPT Agent (OpenAI) — Not on Squarespace’s list: https://getcitable.in/crawlers/chatgpt-agent/squarespace
• Operator (OpenAI) — Not on Squarespace’s list: https://getcitable.in/crawlers/operator/squarespace
• Gemini-Deep-Research (Google) — Not on Squarespace’s list: https://getcitable.in/crawlers/gemini-deep-research/squarespace
• Google-Agent (Google) — Not on Squarespace’s list: https://getcitable.in/crawlers/google-agent/squarespace
• NotebookLM (Google) — Not on Squarespace’s list: https://getcitable.in/crawlers/notebooklm/squarespace
• Kimi-User (Moonshot AI) — Not on Squarespace’s list: https://getcitable.in/crawlers/kimi-user/squarespace
• TongyiBot (Alibaba) — Not on Squarespace’s list: https://getcitable.in/crawlers/tongyibot/squarespace
• YiyanBot (Baidu) — Not on Squarespace’s list: https://getcitable.in/crawlers/yiyanbot/squarespace
• DoubaoBot (ByteDance) — Not on Squarespace’s list: https://getcitable.in/crawlers/doubaobot/squarespace
• Amzn-User (Amazon) — Not on Squarespace’s list: https://getcitable.in/crawlers/amzn-user/squarespace
• AmazonBuyForMe (Amazon) — Not on Squarespace’s list: https://getcitable.in/crawlers/amazonbuyforme/squarespace
• NovaAct (Amazon) — Not on Squarespace’s list: https://getcitable.in/crawlers/novaact/squarespace
• amazon-QBusiness (Amazon) — Not on Squarespace’s list: https://getcitable.in/crawlers/amazon-qbusiness/squarespace
• kagi-fetcher (Kagi) — Not on Squarespace’s list: https://getcitable.in/crawlers/kagi-fetcher/squarespace
• PhindBot (Phind) — Not on Squarespace’s list: https://getcitable.in/crawlers/phindbot/squarespace
• LinerBot (Liner) — Not on Squarespace’s list: https://getcitable.in/crawlers/linerbot/squarespace
• Manus-User (Manus) — Not on Squarespace’s list: https://getcitable.in/crawlers/manus-user/squarespace
• TwinAgent (Twin) — Not on Squarespace’s list: https://getcitable.in/crawlers/twinagent/squarespace
• ShapBot (Parallel) — Not on Squarespace’s list: https://getcitable.in/crawlers/shapbot/squarespace
• bigsur.ai (Big Sur AI) — Not on Squarespace’s list: https://getcitable.in/crawlers/bigsur-ai/squarespace
• QualifiedBot (Qualified) — Not on Squarespace’s list: https://getcitable.in/crawlers/qualifiedbot/squarespace
• KlaviyoAIBot (Klaviyo) — Not on Squarespace’s list: https://getcitable.in/crawlers/klaviyoaibot/squarespace
• Poggio-Citations (Poggio) — Not on Squarespace’s list: https://getcitable.in/crawlers/poggio-citations/squarespace
• iAskBot (iAsk) — Not on Squarespace’s list: https://getcitable.in/crawlers/iaskbot/squarespace
• Crawl4AI (Crawl4AI) — Not on Squarespace’s list: https://getcitable.in/crawlers/crawl4ai/squarespace
• Crawlspace (Crawlspace) — Not on Squarespace’s list: https://getcitable.in/crawlers/crawlspace/squarespace
• WRTNBot (WRTN) — Not on Squarespace’s list: https://getcitable.in/crawlers/wrtnbot/squarespace
• UseAI (UseAI) — Not on Squarespace’s list: https://getcitable.in/crawlers/useai/squarespace
• AIWebIndex (Lyrenth) — Not on Squarespace’s list: https://getcitable.in/crawlers/aiwebindex/squarespace
• Aranet-SearchBot (Aranet) — Not on Squarespace’s list: https://getcitable.in/crawlers/aranet-searchbot/squarespace
• KunatoCrawler (Kunato) — Not on Squarespace’s list: https://getcitable.in/crawlers/kunatocrawler/squarespace
• Reflectionbot (Reflection) — Not on Squarespace’s list: https://getcitable.in/crawlers/reflectionbot/squarespace
• GeistHaus-PageFetcher (GeistHaus) — Not on Squarespace’s list: https://getcitable.in/crawlers/geisthaus-pagefetcher/squarespace
• MistralAI-Index (Mistral) — Not on Squarespace’s list: https://getcitable.in/crawlers/mistralai-index/squarespace
• Kimi-SearchBot (Moonshot AI) — Not on Squarespace’s list: https://getcitable.in/crawlers/kimi-searchbot/squarespace
• meta-webindexer (Meta) — Not on Squarespace’s list: https://getcitable.in/crawlers/meta-webindexer/squarespace
• Bravebot (Brave) — Not on Squarespace’s list: https://getcitable.in/crawlers/bravebot/squarespace
• ExaBot (Exa) — Not on Squarespace’s list: https://getcitable.in/crawlers/exabot/squarespace
• TavilyBot (Tavily) — Not on Squarespace’s list: https://getcitable.in/crawlers/tavilybot/squarespace
• AddSearchBot (AddSearch) — Not on Squarespace’s list: https://getcitable.in/crawlers/addsearchbot/squarespace
• LinkupBot (Linkup) — Not on Squarespace’s list: https://getcitable.in/crawlers/linkupbot/squarespace
• QueritBot (Querit) — Not on Squarespace’s list: https://getcitable.in/crawlers/queritbot/squarespace
• TerraCotta (Ceramic AI) — Not on Squarespace’s list: https://getcitable.in/crawlers/terracotta/squarespace
• HenkBot (Valyu) — Not on Squarespace’s list: https://getcitable.in/crawlers/henkbot/squarespace
• Channel3Bot (Channel3) — Not on Squarespace’s list: https://getcitable.in/crawlers/channel3bot/squarespace
• AzureAI-SearchBot (Microsoft) — Not on Squarespace’s list: https://getcitable.in/crawlers/azureai-searchbot/squarespace
• Amzn-SearchBot (Amazon) — Not on Squarespace’s list: https://getcitable.in/crawlers/amzn-searchbot/squarespace
• atlassian-bot (Atlassian) — Not on Squarespace’s list: https://getcitable.in/crawlers/atlassian-bot/squarespace
• Cloudflare-AutoRAG (Cloudflare) — Not on Squarespace’s list: https://getcitable.in/crawlers/cloudflare-autorag/squarespace
• Andibot (Andi) — Not on Squarespace’s list: https://getcitable.in/crawlers/andibot/squarespace
• Anomura (Direqt) — Not on Squarespace’s list: https://getcitable.in/crawlers/anomura/squarespace
• YandexAdditional (Yandex) — Not on Squarespace’s list: https://getcitable.in/crawlers/yandexadditional/squarespace
• GoogleOther (Google) — Covered by the checkbox: https://getcitable.in/crawlers/googleother/squarespace
• ZanistaBot (Zanista) — Not on Squarespace’s list: https://getcitable.in/crawlers/zanistabot/squarespace
• Poseidon Research Crawler (Poseidon Research) — Not on Squarespace’s list: https://getcitable.in/crawlers/poseidon-research-crawler/squarespace

Checked against Squarespace's own documentation on 1 October 2026

• Squarespace Help Center: Request that AI models exclude your site: https://support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site

About Citable

Citable strips the UI from a website for AI agents and serves them only the content.

When an AI agent — ChatGPT, Claude, Perplexity or any of the 81 that Citable recognises — opens a page, Citable removes the theme, scripts, navigation and layout and serves only the content: the same facts, at the same URL, on the site's own domain. People and Google still get the full designed page.

How it works: https://getcitable.in/how-it-works · Summary for AI agents: https://getcitable.in/llms.txt

Source: https://getcitable.in/crawlers/on/squarespace · Citable · Run the free audit · llms.txt · getcitable@gmail.com