Hello friends, today I wanted to talk a little more about Sitecore’s recent acquisition, Scrunch. Specifically the technology, which really got me thinking about how AI is already shifting the landscape. There’s a statistic I heard about a month ago that still sticks with me: there are more AI bots interacting with your website than humans. That made me pause and wonder, will humans visiting websites die out in the future? Will people just interact with a Claude Desktop–style application to do everything they need online? Possibly. But I also find that I still need to search the web directly to get answers, because AI doesn’t always interpret pages correctly.

That’s where Scrunch adds value. Not only do you now need to pay attention to AI bots viewing your site, there are also different types of AI agents visiting your website, and that requires a shift in how you surface data to those users.

The AI Agents

Like I just mentioned there are many different types of AI agents or bots landing on your website at any given moment. These agents fit into three categories listed below, and Scrunch AI has also written more details about this topic: Understanding AI Bot Types (Retrieval, Indexer, Training, Uncategorized) | Scrunch Help Center.

Training data collectors

As the name suggests, these agents collect content that may be used for model training, but not as raw, unfiltered captures. In practice, pipelines typically include steps to strip or reduce user-specific data and remove generic page chrome before the content is stored or processed. Conceptually, it’s similar to the old Google crawler used for SEO, except the goal is to feed information into an AI system’s broader knowledge base.

Importantly, seeing this traffic doesn’t automatically mean your site is being used to train a model, it often just means bots are visiting with that intent. You’ll frequently spot them via Bot style user agent strings because they’re not triggered by an individual user action, but rather they’re automated processes scanning the web.

These requests are usually server-side only: the bot fetches HTML and doesn’t execute the JavaScript-driven interactivity meant for humans. That’s also why it can make sense to tailor an Agent Experience Platform (AXP) toward these agents, to remove unnecessary UI context, surface the core information more directly, and reduce compute costs for both sides.

Retrieval-time fetchers (RAG)

These are usually triggered by a user. I’ve often had Claude Desktop research a topic—for example, navigating to a page to collect information or conducting a web search—then retrieving relevant results, reading the returned data, and using that information to produce a more in-depth response. The user agents for this type of bot are often prefixed with -User, indicating that a user is waiting for the request to complete.

These bots may also sometimes skip reading the robots.txt, since they can be perceived as almost human. This is another use case where these agents hit the page server-side only. In this situation, it can make a lot of sense to strip out unnecessary CSS or JavaScript and output the core topic of the site so an LLM can understand the page more easily.

Agentic Browsers

This is a category I haven’t explored as much yet: agentic browsers like the Claude browser extension or Perplexity Comet. Unlike “fetcher” bots that pull HTML server-side, these tools operate alongside a human in a real browser session. That means they often render the page and may interact with it, aka loading CSS and JavaScript, following links, clicking buttons, and reading the UI much more like a person would. Because of that, they generate client-side signals (and can be observed by client-side analytics), but they can also behave differently than humans (faster navigation, unusual interaction patterns, higher request volume).

How Can You Track?

As I’ve alluded to throughout the agent types above, it would be great to have clear visibility into this traffic, but most analytics stacks are client-side (e.g., Google Analytics, Sitecore AI). That means you often won’t see the agents that simply request HTML and never execute your JavaScript. This is the key observability gap: server-side fetchers can consume your content without ever showing up in your usual dashboards.

Scrunch’s approach (and most “agent traffic” solutions) starts at the edge/CDN, where every request is visible. From there you can:

• Identify likely agent classes from headers + IP reputation + behavior patterns
• Separate human vs agent traffic in reporting
• Route agents to a simplified, machine-readable experience (AXP) while keeping the human experience intact

Scrunch has a good starting point on this in their docs: Agent Traffic | Scrunch Help Center.

If you wanted to track this traffic without Scrunch AI, you can get surprisingly far with a server-side interception point in your own stack. Or better yet, go into CloudFlare and create an edge worker to track this data. You could write a Next.js middleware, however if you have a CDN in place like Cloudflare, then you won’t necessarily see all the traffic depending on what is cached and served from Cloudflare directly.

Edge worker example (Cloudflare Worker)

If you control your CDN (or can run logic at the edge), an Edge Worker gives you the same “intercept every request” capability before caching, routing, and origin fetch. That’s why this layer is typically the most reliable place to measure bot/agent traffic.

This example:

• classifies requests from the User-Agent
• captures path + IP
• emits an event to an analytics endpoint (or you can write to a queue / logs / KV)
• optionally routes agents to an alternate experience (e.g., ?axp=1 or a different origin)

Copy
1// Cloudflare Worker (module syntax)
2// wrangler.toml should define:
3// - AGENT_TRAFFIC_ENDPOINT (string)
4// - AGENT_TRAFFIC_KEY (secret)
5
6type AgentType = "training" | "index" | "retrieval" | "agentic" | "unknown"
7
8type Env = {
9	AGENT_TRAFFIC_ENDPOINT?: string
10	AGENT_TRAFFIC_KEY?: string
11}
12
13// Ordered most-specific first. Retrieval must be checked before training:
14// Claude-User's UA contains "anthropic.com" and would otherwise match ClaudeBot's rule.
15const AGENT_SIGNATURES: Array<[AgentType, RegExp]> = [
16	// User-triggered fetchers — a human is waiting on the response
17	["retrieval", /(chatgpt-user|claude-user|perplexity-user|google-agent|duckassistbot|mistralai-user|meta-externalfetcher)/],
18
19	// Search/answer index builders
20	["index", /(oai-searchbot|claude-searchbot|perplexitybot|applebot)/],
21
22	// Training / bulk crawlers
23	["training", /(gptbot|claudebot|ccbot|meta-externalagent|bytespider|amazonbot|diffbot|omgili|timpibot|pangubot|imagesiftbot|cohere)/],
24
25	// Headless automation. Note: this does NOT catch consumer agentic browsers
26	// (Comet, Atlas, Chrome auto browse) — those present as ordinary Chrome.
27	["agentic", /(headless|playwright|puppeteer|selenium|browserbase)/],
28]
29
30function classifyAgent(userAgent: string | null): AgentType {
31	const ua = (userAgent ?? "").toLowerCase()
32	if (!ua) return "unknown"
33
34	for (const [type, pattern] of AGENT_SIGNATURES) {
35		if (pattern.test(ua)) return type
36	}
37
38	// Convention: a "-User" suffix means user-triggered. Catches new entrants
39	// before you've added them to the list above.
40	if (/-user\b/.test(ua)) return "retrieval"
41
42	return "unknown"
43}
44
45function shouldLog(agentType: AgentType, userAgent: string | null): boolean {
46	if (agentType !== "unknown") return true
47	return /bot|crawler|spider/i.test(userAgent ?? "")
48}
49
50async function emitEvent(env: Env, event: Record<string, unknown>) {
51	if (!env.AGENT_TRAFFIC_ENDPOINT || !env.AGENT_TRAFFIC_KEY) return
52
53	// Fire-and-forget: don't block the response path.
54	fetch(env.AGENT_TRAFFIC_ENDPOINT, {
55		method: "POST",
56		headers: {
57			"content-type": "application/json",
58			authorization: `Bearer ${env.AGENT_TRAFFIC_KEY}`,
59		},
60		body: JSON.stringify(event),
61	}).catch(() => {})
62}
63
64export default {
65	async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
66		const ua = request.headers.get("user-agent")
67		const agentType = classifyAgent(ua)
68		const url = new URL(request.url)
69
70		// Cloudflare-specific client IP header (best-effort)
71		const ip = request.headers.get("cf-connecting-ip") ?? undefined
72
73		// UA strings are free text. Cloudflare verifies known crawlers by reverse DNS
74		// and published IP ranges. A claimed agent type with a null verification is
75		// worth flagging — that's likely spoofed traffic.
76		const cf = (request as any).cf
77		const verifiedBot = cf?.verifiedBotCategory ?? null
78
79		if (shouldLog(agentType, ua)) {
80			ctx.waitUntil(
81				emitEvent(env, {
82					ts: new Date().toISOString(),
83					agentType,
84					verifiedBot,
85					ua,
86					ip,
87					path: url.pathname,
88					method: request.method,
89					accept: request.headers.get("accept"),
90				}),
91			)
92		}
93
94		// Optional: route agents to an AXP variant (simple example)
95		// if (agentType !== "unknown") {
96		// 	url.searchParams.set("axp", "1")
97		// 	return fetch(new Request(url.toString(), request))
98		// }
99
100		return fetch(request)
101	},
102}

This is just an example, and gives some indication why having Scrunch with the technology already in place makes a lot of sense for Sitecore. This also probably explains why some Marketers might be surprised to learn there is all this additional AI traffic, it’s because the common client side tracking tools are just not tracking that traffic. The example above assumes you have an API to send the data, which is a pretty vague approximation of an approach to achieve the outcome you are looking for.

What is AXP?

What exactly is the Agent Experience Platform (AXP)? Well it’s the concept that expands on the earlier knowledge that I shared on the use of Agents visiting your website. We want to make sure that the experience for the agent is personalized to that specific agents purpose.

In Scrunch’s case, an AXP is essentially a parallel delivery layer that sits at (or is routed to from) the edge/CDN: Scrunch detects that the visitor is an AI bot (training, retrieval-time, or agentic), classifies the intent, then serves a streamlined, machine-readable version of the page that prioritizes the core content and metadata over the full human UI. Instead of shipping heavy client-side JavaScript, tracking pixels, and complex layout chrome, the AXP response can include clean HTML (or other structured formats), clearer content hierarchy, and bot-friendly signals (e.g., canonical info, entity/FAQ blocks, product details), while still respecting rules like access controls and robots directives. The end result is that agents get “the right data for the job” with less noise and cost, and humans still see the full experience.

Further Out: Discoverable MCP Servers

I’ve been thinking about AXP and how I’m not really sure that is the perfect solve. It relies on just returning information based on an existing human concept of webpages on a website. What we really need is a way for AI to crawl and search for specific content it needs, not end up on a web page that then just slims down the page for AI consumption. There is a different model sitting just over the horizon, and I think it’s where this will eventually land.

From document delivery to interface delivery

If you've built anything with MCP, you already know the shape of this. You define tools, the client discovers them, the model calls them with typed arguments, and you return structured data. Nothing about that is new. What's new is the idea that a website could advertise an MCP server the same way it advertises a sitemap.xml today, and that an agent showing up cold could find it without anyone configuring anything.

That's the piece that's been missing, and it's actively being worked on. SEP-1649 proposes MCP Server Cards: a .well-known/mcp.json endpoint that lets a client discover a server's capabilities, transports, auth requirements, and tool descriptions before establishing a connection. It's on the official MCP 2026 roadmap explicitly so that browsers, crawlers, and registries can inspect what a server does without opening a full session. There's even an IETF draft floating around for an mcp: URI scheme and domain-level discovery.

Practically, that card is just a small JSON file. The spec is still in draft so the exact shape will move, but it's roughly this:

Copy
1{
2  "name": "acme",
3  "description": "Product documentation, pricing, and availability for Acme",
4  "endpoint": "https://acme.com/mcp",
5  "transport": "streamable-http",
6  "tools": [
7    { "name": "search_docs", "description": "Search product documentation" },
8    { "name": "get_pricing", "description": "Current pricing by plan and region" },
9    { "name": "check_availability", "description": "Live inventory by SKU and location" }
10  ]
11}

An agent hits your domain, reads that, and now it knows to call get_pricing instead of guessing at a pricing page. No layout to misread, no chrome to strip, no AXP round trip. And notice that last tool, live inventory is something a document fundamentally can't express, because it's only true at the moment it rendered.

The catch is that nothing calls this yet. GPTBot and friends still issue a plain GET and move on. So this is a bet on where things land in twelve to twenty-four months, not something you ship next sprint, which is exactly why Scrunch and AXP make sense right now. But it doesn’t hurt to start dreaming about where exposing tools or distinct RAG sources to your Agent users will make a ton of sense.