Your agent's fetch tool does not render JavaScript. Most SMB stacks pay for that twice.
Once when a quiet agent silently misses everything on a single-page app, returns an empty shell, and ships a confidently wrong answer built on no content. And once more when the team panics and reaches for a real browser on every URL, including the seventy percent that never needed one. The fix is a routing layer with three signals and three tiers. This is what I install for SMB clients before any of the headline AI work begins.
Direct answer, verified 2026-05-04 against platform.claude.com web_fetch docs
When does an agent need a real browser to fetch?
When the URL returns content that was rendered by JavaScript on the client. Anthropic's web_fetch tool, the default a Claude agent uses when you hand it a URL, performs a server-side HTTP fetch and reads the response. It does not execute JavaScript, does not boot a browser, does not wait for hydration. The official documentation says it plainly:
“The web fetch tool currently does not support websites dynamically rendered via JavaScript.”
On a modern SPA (React, Next.js with client components, Vue, Svelte) that response is an empty mount node and a bundle script tag. None of the page content is in the body. A real browser render boots Chromium, runs the bundle, and snapshots the DOM after hydration. That is the only way to see what a human user sees on those pages. The decision rule for the rest of this guide is: detect when the cheap fetch is enough, escalate only when it is not, and never run a browser on a URL where a fetch already told you everything.
What an agent fetch actually returns on a SPA
A short live example so the failure mode is unambiguous. The terminal below is what an agent receives back from web_fetch on a typical client-rendered marketing page. The body is short. The mount node is empty. The bundle is named. The agent then summarizes that as if it were content, and the client gets a polished paragraph that has no relationship to what a user opening the page would see.
The dangerous part is not that the fetch returned an empty body, it is that the agent kept going. Without a routing layer, that result looks like a successful fetch and the rest of the agent loop reads it as content. The fix is detection before generation, not bigger models.
The three signals that say “escalate this URL”
Three checks, ordered cheapest first, that catch the cases where a fetch is incomplete. Run them on every fetch result before letting the agent reason over the body. Anything that trips a signal goes to the next tier; everything else passes through and the agent never knows there was a router.
“The web fetch tool currently does not support websites dynamically rendered via JavaScript.”
platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool, retrieved 2026-05-04
// Three signals that say "the fetch result is incomplete,
// escalate this URL to a real browser render."
type FetchResult = {
status: number;
headers: Record<string, string>;
body: string;
};
export function needsBrowserRender(r: FetchResult): boolean {
// 1. Cloudflare JavaScript challenge.
// The cf-ray header on a 403 (or a 503) is the giveaway.
if (
(r.status === 403 || r.status === 503) &&
r.headers["cf-ray"] !== undefined
) {
return true;
}
if (r.body.includes("checking your browser")) return true;
// 2. SPA mount-node shell, no children rendered yet.
// React, Next, Vue, Svelte all leave a mount node like this.
const mountNodeShell =
/<div id="(?:root|__next|app|svelte)"\s*>\s*<\/div>/i;
if (mountNodeShell.test(r.body)) return true;
// 3. Body too small to be the page the user sees.
// Strip head + scripts + comments, count what's left.
const visible = r.body
.replace(/<head[\s\S]*?<\/head>/gi, "")
.replace(/<script[\s\S]*?<\/script>/gi, "")
.replace(/<!--[\s\S]*?-->/g, "")
.replace(/\s+/g, " ")
.trim();
if (visible.length < 2_048) return true;
return false;
}The Cloudflare check is first because it is the only signal that requires a different tier than the SPA case. A SPA shell can be cleared with a plain headless Chromium. A Cloudflare challenge generally cannot. Sending the wrong tier wastes a render and still fails. The mount-node check is the most common trigger on SMB workloads (most marketing sites are now React or Next), and the body-length check is the safety net that catches edge cases the first two miss.
The three-tier ladder
Same shape every time. The fetch tool tries first, and only the URLs that fail the detection check escalate. On a representative SMB monitoring workload (supplier price pages, listing analyses, intake form scrapes, public order status pages), 60 to 80 percent of URLs clear at tier one. The remainder split between tier two and tier three depending on whether the failure looked like a client render or a bot challenge.
agent fetch escalation
Tier 1: plain fetch
200-500 ms, no JS, ~$0 / 10K URLs
detect
cf-ray? mount shell? body < 2KB?
Tier 2: headless render
1-4 s cold, ~$50-100 / 10K URLs
detect
still blocked? still empty?
Tier 3: stealth render
patched browser, ~$150-300 / 10K URLs
// Three-tier ladder. Each call goes through this gate.
// The cheap tier wins on 60 to 80 percent of SMB URLs.
import { needsBrowserRender } from "./detect";
export async function fetchForAgent(url: string): Promise<string> {
// TIER 1: plain HTTP fetch.
// 200 to 500 ms, negligible compute, no JS.
const r1 = await plainFetch(url);
if (!needsBrowserRender(r1)) return r1.body;
// TIER 2: headless Chromium.
// 1 to 4 s cold, 200 to 800 ms warm, JS executed, hydration awaited.
// Skip if the failure was a Cloudflare challenge, that needs tier 3.
if (r1.headers["cf-ray"] === undefined) {
const r2 = await headlessRender(url);
if (!isStealthRequired(r2)) return r2.body;
}
// TIER 3: stealth render.
// Patched Chromium with TLS / fingerprint masking. Most expensive.
const r3 = await stealthRender(url);
return r3.body;
}Cost ladder, ten thousand URLs
Numbers help. The table is the per-tier cost on a workload that I scope often (an agent that walks ten thousand supplier or listing URLs per run, on a daily cron). Provider prices below are public list rates from May 2026. The right column is the bill if you skip the router and call the most expensive tier on every URL, which is what most stacks default to.
| Tier | Per page | 10K URLs (this tier only) | Wall-clock, parallel |
|---|---|---|---|
Plain HTTP fetch web_fetch, requests, fetch | $0.00001 | $0.10 plus egress | 3-8 minutes |
Headless Chromium Playwright, self-hosted, or Browserbase basic | $0.005-0.010 | $50-100 | 30-90 minutes |
Stealth render Browserbase stealth, Cloudflare Browser Run, Obscura, Bright Data | $0.015-0.030 | $150-300 | 1-3 hours |
Routed (60% / 30% / 10%) tier 1, then 2, then 3 only on Cloudflare | blended | $30-60 | 15-40 minutes |
Stealth on every URL (no router) the default most stacks ship with | $0.015-0.030 | $150-300 | 1-3 hours |
The router pays for itself the first day on any workload above a few hundred URLs. The compounding part is wall-clock: on a daily cron, the routed pipeline finishes inside the morning meeting and the no-router pipeline is still running at lunch.
Four questions I ask before reaching for Playwright
On a scoping call, before any code, these are the four questions that decide whether the engagement needs a real browser at all and which tier the heaviest URLs actually sit on. Most engagements stop at question two with a tier-one-and-two solution and never need the stealth tier.
Are the target URLs server-rendered?
Open one in a private window, view source, and look for the actual content in the initial HTML response. If it is there, tier one is enough and the rest of the ladder is overhead. Marketing sites built on Astro, Next.js with server components, or any static site generator usually qualify. SPAs with a single empty mount node do not.
Do the URLs sit behind a JavaScript bot challenge?
Curl the URL with a normal user agent. A 403 with a cf-ray header, a body containing 'checking your browser', or a redirect to a challenge page means tier two is not enough either. Plan for tier three on those URLs, and budget the higher per-page cost into the engagement quote up front.
How many URLs per run, and how often?
Volume decides whether the router is worth installing at all. Below a few hundred URLs per run on a weekly cadence, just use Playwright everywhere and pay $5 to $10 per run. Above a few thousand URLs per day, the router pays back inside a week of operation and the wall-clock improvement is the part that matters most.
Is content needed, or just a screenshot?
Some workflows do not need extracted text at all, they need a visual diff (price tags moved, layout changed, banner appeared). For those, the question is not fetch vs render, it is which screenshot service is cheapest at the volume. The router still applies but the second tier is a screenshot endpoint, not a Playwright session, and the cost ladder shifts.
Where this fits in a c0nsl engagement
The router is rarely the headline deliverable. It usually shows up as the load-bearing infrastructure underneath a customer-facing AI workflow: a supplier price monitor, a listing analyzer, a public order tracking summarizer, an intake-form auto-classifier. The client booked the consult to ship the workflow; the router is what makes the workflow run inside the budget and the SLA they need.
Mapped to the published rates on c0nsl.com: a small integration ($500 to $2,000) installs the three-tier ladder on one workload. A custom system ($2,000 to $10,000+) handles a multi-agent rollout, a shared browser pool, eval cases for the detection signals, and a bill-watching dashboard so the mix between tiers is visible to the founder. The retainer ($1,000 to $5,000 per month) is right when the volume crosses a few hundred thousand monthly URLs and the bot-defense landscape needs ongoing maintenance. The $75 consult is where I look at the actual URLs, run the detection check on a sample, and tell you which tier the engagement should sit at, in writing.
Run the detection check on your actual URLs
Bring a sample of the URLs your agent is supposed to fetch. I run the three-signal check live on the call, sort them into the three tiers, and quote a fixed scope against the published rates so you can see the routed bill before we ship anything.
Frequently asked questions
Does Anthropic's web_fetch tool render JavaScript?
No. The official Anthropic documentation for the web_fetch tool states verbatim: 'The web fetch tool currently does not support websites dynamically rendered via JavaScript.' This means an agent that calls web_fetch on a single-page application built with React, Vue, Svelte, or any other client-side framework receives only the server-rendered shell, typically an empty body with a root div and the bundle script tags. None of the actual content is in that response. The same is true for plain HTTP fetchers (Python requests, Node fetch, curl) on the same URLs.
What is the actual difference between an agent fetch and a real browser render?
An agent fetch is a single HTTP request that returns whatever the origin server sends back. If the server already rendered the page (server-side rendering, static export, or a CDN-cached HTML response), that response contains the full content and a fetch is enough. A real browser render boots a Chromium process, navigates to the URL, executes the JavaScript bundle, waits for hydration and any in-flight network calls, and only then snapshots the DOM. The browser sees what a human user sees. The fetch sees what the origin sent. On a modern SPA, those two answers are completely different.
How do I detect when a fetch result is incomplete and a browser render is needed?
Three signals reliably catch the cases that need escalation. First, body length under roughly two kilobytes after stripping head and script tags, on a page that should be content-heavy, is almost always a SPA shell. Second, the markup contains a single empty mount node like <div id="root"></div> or <div id="__next"></div> with no children, which is what a hydration target looks like before JavaScript runs. Third, an HTTP 403 paired with a cf-ray response header (or a body containing the phrase 'checking your browser') is Cloudflare's JavaScript bot challenge and a plain fetch will never pass it. A short detection function checks all three and routes only the matching URLs to the browser tier.
What is the cost difference per ten thousand pages?
On a representative SMB monitoring workload (supplier price pages, listing analyses, intake form scrapes), a plain HTTP fetch runs in roughly 200 to 500 milliseconds and uses negligible compute, so ten thousand pages cost cents in compute and a few dollars in egress. A headless Chromium render, depending on provider, is roughly $0.005 to $0.03 per page, so the same ten thousand pages cost $50 to $300 plus a wall-clock time of two to ten hours. The router decision is therefore not a small optimization, it is the difference between a $5 job and a $300 job. On the workloads I scope for clients, the right routing typically sends 60 to 80 percent of URLs to the cheap fetch tier and only the remainder to the browser tier.
Can I just use a real browser everywhere and skip the routing layer?
You can, and a lot of stacks do, which is why the bills get out of hand. The trouble is not the per-page price alone, it is also the wall-clock time and the per-session memory cost. A Chromium session holds roughly 150 to 300 megabytes of RAM, takes one to four seconds to boot cold, and serializes badly. A monitoring agent that walks ten thousand URLs through a single browser is going to take hours. The same agent with a routing layer that sends 70 percent of URLs to plain fetch in parallel and only the remaining 30 percent through the browser pool finishes in a fraction of the time and a fraction of the cost. The routing layer is not optional infrastructure once the workload is real.
When does a stealth browser (Cloudflare-aware, fingerprint-randomized) actually matter vs a plain headless browser?
A plain headless Chromium clears most JS-only renders. It does not clear modern bot challenges. By 2026, providers like Cloudflare, DataDome, and Akamai are doing TLS handshake fingerprinting (JA4), low-level Chrome property checks, and behavioral analysis. A vanilla Playwright session announces itself in the headers and the navigator object and gets blocked. The third tier in the ladder is a stealth browser, either a patched Chromium build (Obscura, Cloudflare Browser Run, Browserbase stealth mode) or a paid render API that fronts that for you. The right move for an SMB consulting engagement is to send only the URLs that fail the previous two tiers to this tier, because the per-render cost is the highest of the three.
How does this routing decision tie into Claude Code or the Claude Agent SDK?
Claude Code and the Claude Agent SDK both wire through the Anthropic API, which means the default fetch tool an agent reaches for is web_fetch with the JS limitation above. To wire the three-tier ladder into a Claude agent, register two MCP tools alongside web_fetch: a real-browser tool (the official Playwright MCP server is the easiest path) and a stealth tool (a Browserbase, Cloudflare Browser Run, or Obscura wrapper). Then write a small system prompt that tells the agent the routing rule: try web_fetch first, escalate if the result looks like a SPA shell or a Cloudflare challenge, and only escalate to stealth if the second tier also fails. The agent picks the right tool per URL and the bill stops being a surprise.
What does a fixed-scope engagement for this kind of routing layer cost?
On the published c0nsl rates, a small integration tier ($500 to $2,000) is sized for installing the three-tier ladder on one workload (one agent, one to three thousand URLs per run, one queue, one observability pass). A custom system tier ($2,000 to $10,000+) is sized for multi-agent rollouts with shared browser pools, eval harnesses, and a bill-watching dashboard. A retainer ($1,000 to $5,000 per month) is appropriate when the workload grows past a few hundred thousand monthly URLs and the bot-defense landscape shifts often enough that the third tier needs maintenance. The $75 consult is where I look at the actual URLs and tell you which tier the engagement should sit at.
Where does the Anthropic web_fetch tool's JS limitation come from technically?
The web_fetch tool runs server-side inside Anthropic's infrastructure as a server tool. It performs an HTTP fetch, extracts text content (and base64 PDF content for documents), and feeds the result into the model's context. There is no Chromium process on the server side, no JavaScript execution, no DOM construction. That is why the official documentation says 'currently does not support websites dynamically rendered via JavaScript' rather than 'does not handle some sites,' the limitation is structural to a server-tool fetcher and applies uniformly. The tradeoff is that web_fetch costs nothing beyond the input tokens of the returned content, which is the right answer for the majority of URLs an SMB workload actually touches.
How does the decision change if my client is on OpenAI or Gemini instead of Claude?
The shape of the decision is the same. OpenAI's Responses API ships a hosted web fetcher and a hosted web search; both are HTTP-based and inherit the same JS-rendering limitation. Gemini's URL context tool likewise reads server-rendered content and does not run a browser. The model swap does not change the routing rule, it just changes which native fetch tool sits on the cheap tier. The browser tier and stealth tier are still separate MCP tools or remote services and they look the same regardless of which model is driving the agent. The page on c0nsl.com is written for Claude because that is the default for the SMB stacks I scope, but the architecture transfers.