# Robots.txt for Prospera Healthcare
# https://prosperahealthcare.com
#
# This file is the complete, authoritative robots policy. Cloudflare used to
# prepend its managed "content signals" block (the AI-crawler opt-outs) at the
# edge, and earlier revisions of this file only patched its gaps. Verified
# 2026-09-22: the served robots.txt is byte-identical to this file, so the
# managed block is no longer being added and the full trainer list lives here.
# If the managed block ever reappears, duplicate groups are harmless: RFC 9309
# merges groups for the same user-agent.
#
# /_astro/ is deliberately NOT disallowed. It holds the site's stylesheet, web
# fonts and JS — 15 references on the homepage alone. Blocking it stops Google
# rendering pages the way visitors see them, which is exactly what Google warns
# against. Nothing under /_astro/ is private; the filenames are content-hashed.

# The Content-signal line states the same policy as the per-bot groups below,
# in the Content Signals vocabulary (contentsignals.org): search engines yes,
# AI assistants reading pages to answer a parent's question yes, model
# training no. Keep it consistent with the trainer blocklist below.
User-agent: *
Content-signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://prosperahealthcare.com/sitemap-index.xml

# AI crawlers.
#
# The stance: model TRAINING crawlers are turned away, AI SEARCH and assistant
# crawlers are let in, because those are what put Prospera in an answer when a
# parent asks an assistant about ABA therapy in Texas.
#
# The blocklist below is every major training crawler, applied uniformly, so
# the policy reads as consistent to anything auditing it (Ahrefs flagged the
# old partial list on every indexable page). Do NOT add AI search or assistant
# agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User,
# Claude-User, Claude-SearchBot, Amzn-SearchBot, DuckAssistBot, Applebot,
# Google-Agent, Amazonbot) to this list: blocking those removes the site from
# AI answers without protecting anything.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: cohere-training-data-crawler
Disallow: /

User-agent: Ai2Bot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: PanguBot
Disallow: /

User-agent: Webzio-Extended
Disallow: /

User-agent: Kangaroo Bot
Disallow: /

User-agent: DeepseekBot
Disallow: /

User-agent: xAI-Bot
Disallow: /

# Amazonbot is an AI SEARCH crawler (it feeds Alexa and Rufus answers), not a
# trainer. It stays allowed. The explicit group survives from when Cloudflare's
# managed block disallowed it; per RFC 9309 section 2.2.2 an equal-length ALLOW
# wins the tie, so this is a no-op today and a safety net if the managed block
# returns.

User-agent: Amazonbot
Allow: /