# robots.txt for Deal Insights

User-agent: *
Allow: /

# ---------------------------------------------------------------------------
# AI answer-engine crawlers, explicitly allowed for generative-engine
# optimization (GEO). Two distinct groups:
#
#   1. Retrieval / search bots fetch pages live to cite them in AI answers.
#      These are what let Deal Insights be surfaced as an answer in ChatGPT,
#      Claude, and Perplexity. Highest GEO value - keep these allowed.
#   2. Training crawlers ingest content to train future models. Allowing
#      these builds long-term, baked-in model awareness of Deal Insights.
#
# Note: robots.txt only guides bots that choose to honor it. A Cloudflare
# edge/WAF block takes precedence - a bot blocked there is stopped before it
# ever reaches this file. Keep the Cloudflare AI-bot rules in sync with the
# intent documented here (allow the retrieval bots below).
# ---------------------------------------------------------------------------

# --- Retrieval / search bots (live citation) ---
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# --- Training crawlers (long-term model awareness) ---
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

Sitemap: https://dealinsights.io/sitemap.xml