AI crawlers on Shopify: allow or block GPTBot, ClaudeBot, PerplexityBot and 78 more

Shopify generates robots.txt for every store. It is changed by adding a robots.txt.liquid template to the theme; there is no file to upload. The steps on Shopify, what trips people up there, and a page for each of 81 AI crawlers.

This response

You are ClaudeBot (Anthropic). Citable recognised you as an AI agent, so this is https://getcitable.in/crawlers/on/shopify with the UI removed — the content only. A browser asking for the same address gets the full designed page.

Where Shopify keeps the rule

Shopify generates robots.txt for every store. It is changed by adding a robots.txt.liquid template to the theme; there is no file to upload.

The steps below are shown for GPTBot. Each crawler has its own page, with its own names in the rule.

Step by step

• 1. Open the theme’s code. In the Shopify admin go to Online Store → Themes. On the theme, open the ⋯ menu and choose Edit code.
• 2. Add the robots template. Click Add a new template, choose robots, and create it. This adds templates/robots.txt.liquid, which prints Shopify’s default rules.
• 3. Add the group for GPTBot. Paste the block below the Liquid that prints the default rules, outside any tag. Text there is written into robots.txt as it stands.
• 4. Save, then read it back. Open /robots.txt on the store’s own domain and confirm the group is there.

What to paste

A group that names a crawler replaces the * group for that crawler; it does not add to it. Paths the * group closes are open to a crawler with its own group unless they are repeated there.

Refuse GPTBot:

User-agent: GPTBot
Disallow: /

Allow GPTBot:

User-agent: GPTBot
Allow: /

What trips people up

The template lives in the theme. Publish a different theme and the rule goes with the old one, unless the new theme has the same template.

Asked, or actually refused?

Shopify gives a site owner robots.txt and nothing stronger: there is no setting that refuses a request by its user agent. robots.txt is a request. A crawler that honours it stays out; nothing in the file stops one that does not. What is enforced is what the server or the edge in front of it refuses.

Check it yourself

A 200 means the name is let through; a 403 means something in front of the page refuses it. This tests the name from your own address. A platform that checks a crawler’s address as well may treat the real GPTBot differently.

The request:

curl -I -A "GPTBot" https://your-site.example/

Every AI crawler, on Shopify

• OAI-SearchBot (OpenAI): https://getcitable.in/crawlers/oai-searchbot/shopify
• ChatGPT-User (OpenAI): https://getcitable.in/crawlers/chatgpt-user/shopify
• GPTBot (OpenAI): https://getcitable.in/crawlers/gptbot/shopify
• Claude-SearchBot (Anthropic): https://getcitable.in/crawlers/claude-searchbot/shopify
• Claude-User (Anthropic): https://getcitable.in/crawlers/claude-user/shopify
• Claude-Web (Anthropic): https://getcitable.in/crawlers/claude-web/shopify
• ClaudeBot (Anthropic): https://getcitable.in/crawlers/claudebot/shopify
• Perplexity-User (Perplexity): https://getcitable.in/crawlers/perplexity-user/shopify
• PerplexityBot (Perplexity): https://getcitable.in/crawlers/perplexitybot/shopify
• Google-Extended (Google): https://getcitable.in/crawlers/google-extended/shopify
• meta-externalfetcher (Meta): https://getcitable.in/crawlers/meta-externalfetcher/shopify
• meta-externalagent (Meta): https://getcitable.in/crawlers/meta-externalagent/shopify
• DuckAssistBot (DuckDuckGo): https://getcitable.in/crawlers/duckassistbot/shopify
• CCBot (Common Crawl): https://getcitable.in/crawlers/ccbot/shopify
• Applebot-Extended (Apple): https://getcitable.in/crawlers/applebot-extended/shopify
• Bytespider (ByteDance): https://getcitable.in/crawlers/bytespider/shopify
• Amazonbot (Amazon): https://getcitable.in/crawlers/amazonbot/shopify
• YouBot (You.com): https://getcitable.in/crawlers/youbot/shopify
• cohere-ai (Cohere): https://getcitable.in/crawlers/cohere-ai/shopify
• PetalBot (Huawei): https://getcitable.in/crawlers/petalbot/shopify
• Diffbot (Diffbot): https://getcitable.in/crawlers/diffbot/shopify
• AI2Bot (AI2): https://getcitable.in/crawlers/ai2bot/shopify
• Timpibot (Timpi): https://getcitable.in/crawlers/timpibot/shopify
• Kangaroo Bot (Kangaroo): https://getcitable.in/crawlers/kangaroo-bot/shopify
• omgili (Webz.io): https://getcitable.in/crawlers/omgili/shopify
• MistralAI-User (Mistral): https://getcitable.in/crawlers/mistralai-user/shopify
• ChatGPT Agent (OpenAI): https://getcitable.in/crawlers/chatgpt-agent/shopify
• Operator (OpenAI): https://getcitable.in/crawlers/operator/shopify
• Gemini-Deep-Research (Google): https://getcitable.in/crawlers/gemini-deep-research/shopify
• Google-Agent (Google): https://getcitable.in/crawlers/google-agent/shopify
• NotebookLM (Google): https://getcitable.in/crawlers/notebooklm/shopify
• Kimi-User (Moonshot AI): https://getcitable.in/crawlers/kimi-user/shopify
• TongyiBot (Alibaba): https://getcitable.in/crawlers/tongyibot/shopify
• YiyanBot (Baidu): https://getcitable.in/crawlers/yiyanbot/shopify
• DoubaoBot (ByteDance): https://getcitable.in/crawlers/doubaobot/shopify
• Amzn-User (Amazon): https://getcitable.in/crawlers/amzn-user/shopify
• AmazonBuyForMe (Amazon): https://getcitable.in/crawlers/amazonbuyforme/shopify
• NovaAct (Amazon): https://getcitable.in/crawlers/novaact/shopify
• amazon-QBusiness (Amazon): https://getcitable.in/crawlers/amazon-qbusiness/shopify
• kagi-fetcher (Kagi): https://getcitable.in/crawlers/kagi-fetcher/shopify
• PhindBot (Phind): https://getcitable.in/crawlers/phindbot/shopify
• LinerBot (Liner): https://getcitable.in/crawlers/linerbot/shopify
• Manus-User (Manus): https://getcitable.in/crawlers/manus-user/shopify
• TwinAgent (Twin): https://getcitable.in/crawlers/twinagent/shopify
• ShapBot (Parallel): https://getcitable.in/crawlers/shapbot/shopify
• bigsur.ai (Big Sur AI): https://getcitable.in/crawlers/bigsur-ai/shopify
• QualifiedBot (Qualified): https://getcitable.in/crawlers/qualifiedbot/shopify
• KlaviyoAIBot (Klaviyo): https://getcitable.in/crawlers/klaviyoaibot/shopify
• Poggio-Citations (Poggio): https://getcitable.in/crawlers/poggio-citations/shopify
• iAskBot (iAsk): https://getcitable.in/crawlers/iaskbot/shopify
• Crawl4AI (Crawl4AI): https://getcitable.in/crawlers/crawl4ai/shopify
• Crawlspace (Crawlspace): https://getcitable.in/crawlers/crawlspace/shopify
• WRTNBot (WRTN): https://getcitable.in/crawlers/wrtnbot/shopify
• UseAI (UseAI): https://getcitable.in/crawlers/useai/shopify
• AIWebIndex (Lyrenth): https://getcitable.in/crawlers/aiwebindex/shopify
• Aranet-SearchBot (Aranet): https://getcitable.in/crawlers/aranet-searchbot/shopify
• KunatoCrawler (Kunato): https://getcitable.in/crawlers/kunatocrawler/shopify
• Reflectionbot (Reflection): https://getcitable.in/crawlers/reflectionbot/shopify
• GeistHaus-PageFetcher (GeistHaus): https://getcitable.in/crawlers/geisthaus-pagefetcher/shopify
• MistralAI-Index (Mistral): https://getcitable.in/crawlers/mistralai-index/shopify
• Kimi-SearchBot (Moonshot AI): https://getcitable.in/crawlers/kimi-searchbot/shopify
• meta-webindexer (Meta): https://getcitable.in/crawlers/meta-webindexer/shopify
• Bravebot (Brave): https://getcitable.in/crawlers/bravebot/shopify
• ExaBot (Exa): https://getcitable.in/crawlers/exabot/shopify
• TavilyBot (Tavily): https://getcitable.in/crawlers/tavilybot/shopify
• AddSearchBot (AddSearch): https://getcitable.in/crawlers/addsearchbot/shopify
• LinkupBot (Linkup): https://getcitable.in/crawlers/linkupbot/shopify
• QueritBot (Querit): https://getcitable.in/crawlers/queritbot/shopify
• TerraCotta (Ceramic AI): https://getcitable.in/crawlers/terracotta/shopify
• HenkBot (Valyu): https://getcitable.in/crawlers/henkbot/shopify
• Channel3Bot (Channel3): https://getcitable.in/crawlers/channel3bot/shopify
• AzureAI-SearchBot (Microsoft): https://getcitable.in/crawlers/azureai-searchbot/shopify
• Amzn-SearchBot (Amazon): https://getcitable.in/crawlers/amzn-searchbot/shopify
• atlassian-bot (Atlassian): https://getcitable.in/crawlers/atlassian-bot/shopify
• Cloudflare-AutoRAG (Cloudflare): https://getcitable.in/crawlers/cloudflare-autorag/shopify
• Andibot (Andi): https://getcitable.in/crawlers/andibot/shopify
• Anomura (Direqt): https://getcitable.in/crawlers/anomura/shopify
• YandexAdditional (Yandex): https://getcitable.in/crawlers/yandexadditional/shopify
• GoogleOther (Google): https://getcitable.in/crawlers/googleother/shopify
• ZanistaBot (Zanista): https://getcitable.in/crawlers/zanistabot/shopify
• Poseidon Research Crawler (Poseidon Research): https://getcitable.in/crawlers/poseidon-research-crawler/shopify

Checked against Shopify's own documentation on 1 October 2026

• Shopify Help Center: Editing robots.txt.liquid: https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
• Shopify developer docs: Customize robots.txt: https://shopify.dev/docs/storefronts/themes/seo/robots-txt

About Citable

Citable strips the UI from a website for AI agents and serves them only the content.

When an AI agent — ChatGPT, Claude, Perplexity or any of the 81 that Citable recognises — opens a page, Citable removes the theme, scripts, navigation and layout and serves only the content: the same facts, at the same URL, on the site's own domain. People and Google still get the full designed page.

How it works: https://getcitable.in/how-it-works · Summary for AI agents: https://getcitable.in/llms.txt

Source: https://getcitable.in/crawlers/on/shopify · Citable · Run the free audit · llms.txt · getcitable@gmail.com