The Surfaced crawler

Surfaced analyses how AI answer engines — ChatGPT, Claude, Perplexity, Google AI Overviews — describe a brand, and what on its site makes them get it wrong. When one of your customers (or you) runs a scan, we fetch a small number of public pages from the domain being analysed.

We only ever fetch what a visitor could load without signing in. We do not submit forms, follow links behind authentication, or crawl at a rate a normal browsing session would not produce.

Every request we make carries this header. The name is stable — we will not rename it, because doing so would break firewall rules already deployed:

X-Surfaced-Crawler: https://www.getsurfaced.ai/bot

Almost every request presents a normal browser User-Agent rather than a bot string. That is deliberate, and worth explaining: our requests come from a cloud provider's IP range, and most bot-protection products drop any self-declared bot from a datacenter ASN outright — including on sites whose owners actively want us to read them. The header above is how you tell us apart from a browser.

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36

One path is still the exception, so you will see it in your logs: the first attempt of a scan announces itself honestly as ContentStudio/1.0 (…; GEO Audit Bot) and only retries with the browser string above if it is turned away. It carries the same header. Match the header, not the User-Agent, and both requests are covered by one rule.

If a scan reports that your site refused us, the fix is a rule that skips bot protection for requests carrying our header. In Cloudflare, create a WAF custom rule with action Skip and this expression:

any(http.request.headers["x-surfaced-crawler"][*] == "https://www.getsurfaced.ai/bot")

A header is a Map of arrays in the Cloudflare Rules language, so the comparison has to run through [*] inside any() — a numeric index is not valid there and the editor will reject it. Set the rule to skip Super Bot Fight Mode and any managed rulesets you have enabled. The equivalent exists in AWS WAF (a header-match rule with an ALLOWaction), Akamai, Fastly, and Vercel's firewall.

Custom rules run before Super Bot Fight Mode and the managed rulesets, so a Skip placed there does get us past bot scoring that keys on IP reputation or TLS fingerprint — that scoring is what the rule is skipping. What a header cannot reach is a block that happens before your rules run at all: an IP-level firewall, or a provider whose bot filtering is not expressed as rules you control. If a scan still reports a refusal once the rule is live, send us the Ray ID and we will tell you what we saw.

Our site-analysis crawl reads robots.txt and honours a group aimed at us. Because our User-Agent is a browser string, we match those directives against our product tokens instead — contentstudio, content-studio, getsurfaced, surfaced — any of which addresses us:

User-agent: getsurfaced
Disallow: /

Three caveats, stated plainly rather than discovered later. A robots.txt change only takes effect on our next fetch. And robots.txt currently governs the site-analysis crawl described above — it does not yet govern our other fetches: the sitemap URL a workspace owner enters, the pages we pull to build their knowledge base, the scan someone runs against a domain, or the link check in our site audit. A Disallow on its own will not stop those. We are closing that gap; until it is closed, use the rule below if you need a block that covers everything.

Third: what we act on today is a whole-site Disallow: / (or Disallow: /*) in a group aimed at us. A narrower path rule — Disallow: /members/ — is not yet applied, so please do not read one as keeping us out of that section. To fence off part of a site today, add a URI path condition to the header rule below.

For a block that is immediate and covers every request, invert the rule from the previous section: match the same header and choose Block instead of Skip. Or email us at the address below and we will exclude your domain at our end.

If we are fetching more than you expect, or you want your domain excluded entirely, email info@getsurfaced.ai. If you are debugging a block, include the Cloudflare Ray ID from your security events log — it is the only handle that finds the exact rule that fired.