# ============================================================= # robots.txt for elementalastro.io # Last reviewed: 2026-10-02 (Chinese crawlers refused; see their block below) # 2026-09-01 新增每块一条 `Disallow: /*.html$` —— 挡的是根目录下那 13 个内部预览页 # (设计稿对比 / 分享卡草图 / 无障碍测试台 / 部署后效果预览)。实测那天线上 # /home-redesign-A-prove.html 等四个都是 HTTP 200。里面没有密钥也没有真实用户数据, # 是门面问题。 # 真正的修法有两条,这一条只是兜底: # ① scripts/strip-internal-previews.js 在构建时把它们从 dist 剔掉 —— 新构建里不存在 # ② 已经在服务器上的那一批要工程师手工删(部署若不带 --delete 就会留着) # 两条都做完之后,这 12 行可以撤。 # 注意这条只挡【根目录】的 .html:预渲染那 833 页的 URL 是 /foo/ 形式,不以 .html 结尾, # 不受影响;首页规范 URL 是 /,也不受影响。 # 2026-09-18 每块新增 `Disallow: /email/` —— 提醒邮件里的退订页(/email/unsubscribe),页面本身也是 noindex。 # Strategy: open all public content to search engines and to the AI search and # user-request fetchers (they cite and link us); refuse AI training # crawlers and data scrapers (founder 2026-10-02: protect the content), # keep authenticated user surfaces out of all indexes. # ============================================================= # ─── Default policy: every crawler not explicitly listed below ─── User-agent: * Allow: / Disallow: /payment/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /login Disallow: /register Disallow: /phone-login Disallow: /first-register Disallow: /you-edit Disallow: /cold-fact Disallow: /email/ Disallow: /mirror-answer Disallow: /energy-flashbacks Disallow: /feedback Disallow: /data-deletion Disallow: /install-guide Disallow: /__/ Disallow: /*.html$ # ─── AI / LLM crawlers, each named on purpose ── # Allowed: AI search crawlers and the fetchers that read a page when a # person asks (they cite and link us). Refused: model training crawlers # and data scrapers (founder 2026-10-02: protect the content). robots.txt # is advisory; the edge also refuses the training crawlers by user agent. # OpenAI GPTBot: model training (refused). ChatGPT search uses OAI-SearchBot below. User-agent: GPTBot Disallow: / # OpenAI ChatGPT-User (on-demand fetch when a user pastes a URL) User-agent: ChatGPT-User Allow: / # OpenAI search index crawler User-agent: OAI-SearchBot Allow: / Disallow: /payment/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /login Disallow: /register Disallow: /phone-login Disallow: /first-register Disallow: /you-edit Disallow: /cold-fact Disallow: /email/ Disallow: /mirror-answer Disallow: /energy-flashbacks Disallow: /feedback Disallow: /data-deletion Disallow: /install-guide Disallow: /__/ Disallow: /*.html$ # Anthropic: ClaudeBot trains models; Claude-Web and anthropic-ai are older names (all refused). # Claude-User and Claude-SearchBot are not named here and stay allowed under the default block. User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: anthropic-ai Disallow: / # Perplexity AI User-agent: PerplexityBot Allow: / Disallow: /payment/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /login Disallow: /register Disallow: /phone-login Disallow: /first-register Disallow: /you-edit Disallow: /cold-fact Disallow: /email/ Disallow: /mirror-answer Disallow: /energy-flashbacks Disallow: /feedback Disallow: /data-deletion Disallow: /install-guide Disallow: /__/ Disallow: /*.html$ User-agent: Perplexity-User Allow: / # Google-Extended: Gemini training and grounding token. Allowed (founder 2026-10-02): it lets # the Gemini app cite our pages, and a copier cannot direct it. Google Search uses Googlebot. User-agent: Google-Extended Allow: / Disallow: /payment/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /login Disallow: /register Disallow: /phone-login Disallow: /first-register Disallow: /you-edit Disallow: /cold-fact Disallow: /email/ Disallow: /mirror-answer Disallow: /energy-flashbacks Disallow: /feedback Disallow: /data-deletion Disallow: /install-guide Disallow: /__/ Disallow: /*.html$ # Applebot-Extended: Apple model training token (refused). Applebot itself is not touched. User-agent: Applebot-Extended Disallow: / # CommonCrawl: a public crawl archive anyone can download, used for model training (refused). User-agent: CCBot Disallow: / # Cohere training crawler (refused) User-agent: cohere-ai Disallow: / # Meta AI training crawler (refused). facebookexternalhit, used for link previews, is not touched. User-agent: Meta-ExternalAgent Disallow: / # Amazonbot (Amazon; its crawl improves Amazon products, AI included) and Mistral's training # crawler: refused (founder 2026-10-02: protect the content). User-agent: Amazonbot User-agent: MistralAI-Training Disallow: / # Chinese companies' crawlers: refused (founder 2026-10-02: no crawling from Chinese companies; # copycat sites from Chinese companies target our markets). robots.txt is advisory only; the edge # (Cloudflare) refuses them by user agent and network as well. # Bytespider (ByteDance / Doubao / 抖音 AI search) User-agent: Bytespider Disallow: / # DeepSeek User-agent: DeepSeekBot Disallow: / # Baidu, Sogou, 360, Alibaba Yisou (Shenma), Huawei Petal User-agent: Baiduspider User-agent: Sogou web spider User-agent: 360Spider User-agent: YisouSpider User-agent: PetalBot Disallow: / # Mistral User-agent: MistralAI-User Allow: / # You.com User-agent: YouBot Allow: / # Diffbot: web data extraction service (refused) User-agent: Diffbot Disallow: / # ─── Sitemaps ─── Sitemap: https://www.elementalastro.io/sitemap.xml