# Staged beta rollout: only the pages in server/utils/indexablePaths.ts are
# indexable for now. That's enforced entirely by the X-Robots-Tag layer
# (server/middleware/robots-header.ts), which sends `noindex` on every
# non-allowlisted response — including the crawl itself, so Google actually
# sees the tag. robots.txt stays open (`Allow: /`) on purpose: a `Disallow`
# here would hide pages from crawl but not from the index (an unindexed-but-
# blocked URL can still surface as a bare "No information is available for
# this page" result if something links to it), and it blocked CSS/JS/image
# requests too, breaking rendering in GSC Live Test (TES-28). Don't
# reintroduce a page allowlist here — open the page in indexablePaths.ts
# instead.
#
# Cloudflare's managed robots.txt (Content Signals) was disabled in the
# dashboard on 2026-07-18 (TES-2), so this file is now served as-is. If a
# "# BEGIN Cloudflare Managed content" block ever reappears in the live file,
# someone re-enabled it in Cloudflare AI Crawl Control — turn it back off.
User-agent: *
Allow: /

Sitemap: https://assetfeed.com/sitemap.xml