Stop sending the browser to the model

6 September 2026 · 9 min read · by Akhil Gorantala

Nearly every AI browser tool shipping today shares one architectural decision: to act on a page, it sends a representation of that page (a screenshot, or an accessibility snapshot) to a language model, which reads it and decides what to click. This essay argues that decision is the root cause of the cost, latency and security problems browser agents are known for, that the people hitting those problems are documenting them in public, and that the fix is architectural, not incremental. I build a tool shaped by this argument, so discount accordingly; every load-bearing claim below is linked.

The receipts, from the users themselves

Microsoft's Playwright MCP deliberately chose structured accessibility snapshots over screenshots, and it was the right call inside its architecture: cheaper than vision, more reliable than pixels. But its own issue tracker records what happens on real pages. Issue #1233 asks for a snapshot size cap after captures ranged from 50KB to 540KB, exhausting conversation context "after only a few navigations". Issue #945 measures a single snapshot around 25,000 tokens and proposes preprocessing to strip it. Issue #1247 reports 4.8 million tokens burned in one day debugging a relatively simple automation. These are not competitor attacks; they are users of the most popular tool in the category describing its economics.

Why the loop is the problem

None of those numbers come from bugs. They follow from the loop itself: snapshot, model reads it, model picks a target, action, page changes, snapshot again. The model is the element finder, so every action pays for a model invocation, and every snapshot lands permanently in the conversation context, crowding out the actual work. Vision-based tools (Claude in Chrome and its relatives) run the same loop with screenshots and a vision model, trading token counts for even more latency. Cheaper models shrink the bill but not the round-trips, and nothing shrinks the context accumulation: a long browser session degrades the very conversation it serves.

The arithmetic of a 30-step task

Take a real workflow: update a customer's address in an admin panel, about 30 find-and-act steps. Through a snapshot loop at even a modest 10,000 tokens per step, that is 300,000 tokens of page representations before the model has thought a single useful thought, plus 30 model round-trips of latency, call it a minute of pure waiting. The same task through a local resolver is 30 tool calls of a sentence each, on the order of a thousand tokens total, with each resolution taking single-digit milliseconds. The task's thinking (what to change, which customer, what address) costs the same in both architectures. Everything else is overhead one design carries and the other does not.

The alternative architecture

Split the loop. The model decides what to do: "fill the email field with jo@acme.com". A local engine, inside the browser extension, decides where and how: it ranks the page's real elements from the accessibility tree and DOM against that plain-English intent, in a few milliseconds, with no network and no model, then acts and reports back one line: matched input "Email", confidence 0.98, 7ms. The model never sees the page unless it explicitly asks to read something, and what it reads back is capped and structured rather than a raw dump. This is how Dowser is built, and the design is why its numbers are what they are; the speed is a consequence of the architecture, not the point of it.

When it doesn't know, it doesn't guess

A model reading a snapshot always produces an answer, including when it shouldn't: the second red button probably means delete. A ranker with an explicit confidence score can do something a generative loop structurally cannot: refuse. Below the confidence bar, Dowser returns the ranked candidates ("Delete draft 0.94, Delete account 0.71, Delete attachment 0.43") and acts on none of them, handing the choice back to the model or the human. Deterministic execution is not just cheaper; it is the difference between an agent that misfires quietly and one whose uncertainty is visible and handled.

The security argument nobody markets

There is a second consequence of keeping the page out of the model, and it matters more than the tokens. Playwright MCP issue #1479 describes the risk plainly: accessibility snapshots carry arbitrary page text into model context, where a malicious page's "ignore previous instructions" has a seat at the table. Every snapshot architecture inherits this, and defense means hoping the model can tell data from instructions every single time. A local resolver changes the shape of the problem: the page's content goes into a ranker that outputs structure ({role, name, confidence}), not prose, and what reading the assistant does request comes back capped and labelled as untrusted data. Webpages provide data to the engine; they do not get to speak to your agent. That boundary is architectural, which is the only kind that holds.

FAQ

Isn't this just "Dowser is faster" in more words?

No: the claim is about where the model sits. A snapshot tool with a faster model is still a snapshot tool; the per-action model call and the context accumulation remain. The architecture is the difference, and the speed and cost fall out of it.

When is the snapshot architecture actually better?

Pages with no accessible structure (pure canvas) need vision. Clean-room reproducibility for tests wants Playwright's model regardless of tokens. And if your assistant touches a browser twice a week, none of this arithmetic matters; free snapshot tools are rational there. The full trade-offs are in the category comparison.

Can a local ranker really match a model's flexibility?

For finding named things on structured pages, which is most of what agents do in browsers, yes, and it can prove it with a confidence score. For "click whatever looks most important", a model genuinely reasons better; Dowser's answer is to return candidates and let the model do that reasoning over 50 tokens of structure instead of 10,000 of page.

What about reading pages, not just clicking?

Reading is the one place content must reach the model, so it is capped, structured (an outline or readable text, not raw DOM), and labelled untrusted. Making reads query-shaped, so "order total and shipping status" returns five lines instead of a page, is the next step on our roadmap.

Where do I see this running?

The homepage has a screen recording with live timings, and the setup guide gets any MCP assistant using it in about two minutes.