# yentaknows.ai
#
# Crawlable here: the static pages (/, /connector, /privacy, /terms), the
# directory's text pages (/venues, /cities and each /cities/{city} landing,
# /destinations) and the agent entrypoint /llms.txt. Every text page answers as
# HTML to a browser or crawler and as Markdown to an agent that asks for it.
# Disallowed below are the machine surfaces: /mcp answers POST only, /v1/ is the
# JSON API the MCP tools are a second face of, and the health probes. The one
# exception is /v1/public/ for user-initiated agents (their own group below);
# indexing and training crawlers stay off /v1/ entirely.
#
# Served on the directory host only. Each venue and destination host serves its
# own dynamic /robots.txt that grants every agent the same access as this file
# (one group per agent rather than one per policy, and a host-derived Sitemap:
# line), alongside its own text pages and /sitemap.xml.
#
# /sitemap.xml here is a sitemap index: the directory's own pages
# (/sitemap-pages.xml), every published venue's /sitemap.xml and every published
# destination's /sitemap-pages.xml (a destination's /sitemap.xml is itself an index
# of its listing pages, and an index may not nest another). A Sitemap:
# directive is absolute by spec and a static file cannot interpolate its host,
# so the one below names the apex, which serves /sitemap.xml on every environment
# (the bridge proxies it today, and still will once prod is up); per-host discovery is the
# landing page's <link rel="sitemap"> and the Link: header on the text pages.
#
# Content-Signal (contentsignals.org) states the use preference for the content:
# it is partner-approved for publication, so search, AI input and AI training are
# all "yes". It is a group member, so every group repeats it, and it equals
# `sitemap.CONTENT_SIGNAL`, which the per-tenant robots.txt renders.

User-agent: *
Disallow: /mcp
Disallow: /v1/
Disallow: /healthz
Disallow: /readyz
Content-Signal: search=yes, ai-input=yes, ai-train=yes

# Every named crawler below — traditional search, AI answer-engine and training —
# is welcome to read the public pages; each still skips the machine surfaces above (a
# named group does NOT inherit the "*" group, RFC-9309 §2.2.1, so the disallows are
# repeated rather than assumed). The agents named across the two groups below are
# the canonical `ua.ROBOTS_ALLOW_AGENTS` list the dynamic per-tenant robots.txt
# renders too, and the second group is `ua.ROBOTS_USER_INITIATED_AGENTS`;
# `test_data_sitemap` and `test_robots_user_agents` pin the two surfaces so they
# cannot drift back into separate hand-kept lists or policies.
#
# Every group lists its disallows before `Allow: /`. RFC 9309 picks the longest
# match, so order does not matter to it, but a first-match parser (Python's
# urllib.robotparser, for one) would otherwise stop at `Allow: /`.
User-agent: Googlebot
User-agent: Bingbot
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-Web
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: CCBot
Disallow: /mcp
Disallow: /v1/
Disallow: /healthz
Disallow: /readyz
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes

# User-initiated agents: they fetch because a person asked their assistant a
# question right now, not to index or train. They get the same page access as
# above plus the public REST reads under /v1/public/, which answer the same
# approved data as the pages. Nothing else under /v1/ is opened. The Allow
# comes before the /v1/ disallow so a first-match parser reads it the same way.
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: meta-externalfetcher
Allow: /v1/public/
Disallow: /mcp
Disallow: /v1/
Disallow: /healthz
Disallow: /readyz
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes

# Directs crawlers to the apex sitemap; per-host sitemap discovery is via the
# landing <link rel="sitemap"> and the Link: header.
Sitemap: https://yentaknows.ai/sitemap.xml