# robots.txt for https://yatfa.com
#
# The public site of YATFA. Search engines, AI search, AI assistants and AI training
# crawlers are all welcome: every public page is open to them. Only the API is excluded.
#
# Format: RFC 9309. A crawler obeys only the most specific group that names it and
# ignores the "*" group, so all rules live in the single group below. If you ever add
# a group for one crawler, repeat the Disallow line inside it.
#
# Covered by "*" and allowed on purpose (decision of 2026-10-04):
#   Search:      Googlebot, Bingbot, Applebot, DuckDuckBot
#   AI search:   OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot, DuckAssistBot
#   Assistants:  ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, Meta-ExternalFetcher
#   Training:    GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot
#
# Sign-in pages (/users/...) are deliberately NOT disallowed: they carry a noindex tag,
# and a crawler can only read that tag if it may fetch the page.

User-agent: *
Disallow: /api/
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Content-Usage: search=y, ai-use=y, train-ai=y

# Disallow /api/ - the programmatic API: it needs a key, nothing to index.
# /assets/ stays open: crawlers need styles and scripts to render the pages.
#
# Content-Signal (Cloudflare Content Signals Policy) and Content-Usage (IETF AI Preferences,
# draft-ietf-aipref-attach) state how the content may be used. No major crawler is known
# to act on them yet; crawlers that don't know a line skip it.

# A short Markdown summary of the site for language models and agents:
# https://yatfa.com/llms.txt

Sitemap: https://yatfa.com/sitemap.xml