Vitre - Driving a WebView Instead of Scraping It (Part 1)

Two years ago* I wrote about client-side scraping — doing the crawling and form-filling inside the app, on the user's device, instead of on a rented server driving a browser somewhere. The argument still holds: you spend the client's compute instead of your own, the user never round-trips to a scraping backend, and it is the user's own device signed into the user's own accounts doing the reading, which sidesteps a good deal of the legal awkwardness of scraping someone else's site from a datacenter.

That post ended with "more details will be explained in the coming posts." It took me two years and a rewrite to have details worth explaining. The result is Vitre*, a WebView automation library for Kotlin Multiplatform. You describe what you want done to a page as a list of steps, and it runs them inside an embedded WebView on Android, iOS and the desktop — from your app, from a test, or from an LLM agent. This is the first of three posts about how it works. This one is about the shape of the thing; the second* is about the concurrency model that turned out to be load-bearing; the third* is about handing the page to an agent.

The mess this replaces

Every mobile app already embeds WebViews, and everyone eventually wants to do something to the page inside one: read a value, fill a form, wait for a table to render, talk to the page's own script. The way you do that today is evaluateJavascript with a callback, and the way it goes wrong is always the same.

The calls race each other. A waitFor poll and the extract that depends on it interleave with a reload the user just triggered. They race the UI thread. And none of it survives a port: the Android version is webView.post { } around evaluateJavascript, the iOS version is WKWebView.evaluateJavaScript with subtly different semantics, and the two drift until a boolean reads as true on one platform and "1" on the other — which is a real bug I shipped, not a hypothetical.

Vitre's premise is that a page should be a thing you drive, not a string you interpolate JavaScript into. One step vocabulary, one ordering guarantee, one codebase for every platform.

Why not just run Chrome on a server?

The obvious objection is that headless Chrome already exists. Puppeteer and Playwright drive it well, the ecosystem is mature, and per session they can do things Vitre cannot — full DevTools access, network interception on every platform, request bodies, a dozen tabs at once. If the job is "scrape one site from one place," reach for them and stop reading.

But the use cases from the original post* — aggregating a user's banks, searching their logged-in e-commerce accounts — need one authenticated, personalized browser per user. That is where running Chrome on a server stops being a tooling choice and starts being a business you may not want to be in.

You rent a browser per user. A headless Chrome context is hundreds of megabytes and a core under load, and one isolated logged-in session per active user is a linear cost that grows with your user base, on the most expensive compute you rent. This is exactly the centralized "server hosting instances of client devices" I pointed at two years ago. Vitre spends the browser the user already paid for.

You become the custodian of everyone's session. A server-side browser reading a user's bank means that user's credentials or live cookies sit on your infrastructure. Now you are a breach target and, depending on the data, a regulated custodian — and no amount of scaling fixes the fact that warehousing millions of logins is a liability rather than a feature. Client-side leaves the session on the device where it already is.

You lose the IP arms race. A few thousand users behind one datacenter IP range gets rate-limited, CAPTCHA'd and blocked, because sites treat datacenter traffic as hostile by default. The way out is residential proxy pools, which is another linear cost and a permanent fight. Traffic from the user's own device does not have to look like the user — it is the user, on their own residential IP and real session.

The legal posture is worse. Datacenter scraping a third party is the position courts and terms of service treat most harshly. A user automating their own interaction with a site they are logged into is on different footing.

The one-line version: running the browser server-side makes you rent a browser, an IP, and a liability per user; running it on the client borrows all three from the user for free. What you give up is central control, and on iOS the ability to rewrite a response header — but that trade is the whole reason this library targets the device instead of the datacenter.

Steps, not scripts

The core is a small sealed set of steps: Navigate, LoadHtml, WaitFor, Click, Input, Extract, ExtractRows, Snapshot, EvaluateJs, PostMessage, AwaitMessage. You assemble them with a DSL and hand the list to a WorkflowEngine, which walks it and emits an event per step.

val workflow = workflow("hn-top-story", "Hacker News top story") {
    navigate("https://news.ycombinator.com/")
    waitFor(".titleline > a", timeoutMs = 15_000)
    extract(".titleline > a", into = "headline")
    extract(".titleline > a", into = "url", from = Source.Attribute("href"))
}
 
WorkflowEngine(controller).run(workflow).collect { event ->
    if (event is WorkflowEvent.Completed) println(event.variables["headline"])
}

There is one design decision here that trips people up, so I want to be explicit about it early: the block is a builder, not a script. It runs once, up front, to assemble the step list. Ordinary Kotlin control flow inside it chooses what the workflow contains, but it cannot branch on the page, because when the block runs the page has not been touched yet. An if in there decides which steps exist, not which steps run given what an earlier step found.

That sounds like a limitation, and it is, but it buys two things. A workflow is a value: you can build it, log it, ship it as data, and it is equal to itself. And it is exactly the shape an agent needs, because an agent's steps arrive as JSON rather than as Kotlin — the same WorkflowStep constructors the DSL calls are public, and that is what the MCP server in part three* builds against.

Reading a whole table in one step

The step I reach for most is ExtractRows, because it is the one repeated Extract cannot fake. It returns one JSON record per matching row, with each column resolved within that row.

extractRows(rows = xpath("//li[@data-sku]"), into = "results", limit = 10) {
    // "." is the row itself — read its own attribute.
    column("sku", xpath("."), from = Source.Attribute("data-sku"))
    column("price", xpath(".//span[@class='price']"))
    // Matched on the text a human reads, not on a class hook.
    column("stock", xpath(".//*[normalize-space()='In stock']"))
}

The scoping is the whole point. If you extract prices and SKUs with two independent selectors and zip them, one row missing a price shifts every later price onto the wrong product, silently. Resolving each column inside its row means a missing price is an empty string in that record and nothing else moves.

Three ways to name an element

Every element-addressing step takes a Locator, and there are three kinds.

css("#results .item")      // the common case: short, familiar, fast
xpath("//li[@data-sku]")   // when you need more
handle("e7")               // an element a snapshot already found

A bare string still means CSS, so the shorthand stays short. XPath earns its place where CSS genuinely cannot reach: matching on visible text, walking up the tree with ancestor:: (CSS has no parent combinator), selecting an attribute as a node, positional predicates, count(). Neither pierces shadow DOM — that is the page's doing, not the query language's.

The third kind, handle, is the agent's locator, and it behaves differently on purpose. CSS and XPath describe how to search; a handle names an element the page has already shown you. It is issued by a Snapshot, it lives in the document that issued it, and it dies when that document navigates away. So a handle from the previous page fails loudly instead of quietly resolving against a same-shaped element on the new one. I will come back to snapshots and handles in part three*, because they are what makes an agent possible at all.

Typed reads, so evaluateJs stops lying

evaluateJs returns the JSON encoding of whatever the expression produced, on every platform. That one contract — everything returns JSON — is the thing every platform was made to agree on, and it lets you decode a value instead of string-matching it.

val ready: Boolean = controller.evaluate("document.readyState==='complete'")
val rows: List<Product> = controller.evaluate("Array.from(document.querySelectorAll('li')).map(toRow)")

Getting there was more work than it should have been, because the platforms disagree about the most basic thing. WebKit hands back a Foundation object whose description prints JS true as the string "1"; Android returns JSON "true". So iOS wraps every script in JSON.stringify and the desktop does the same on the way back over CEF's message router, and only then does "true" mean true everywhere. This is the kind of divergence that survives testing for months because both platforms return a value and neither returns an error. The next post* is largely about bugs of exactly this flavour.

Talking to a page you own

If the page is yours — a hybrid app screen, not a third-party site — scraping it is the wrong tool and the bridge is the right one. PostMessage sends a MessageEvent('vitre') into the page; AwaitMessage waits for window.vitre.postMessage coming back. Payloads are @Serializable classes, not hand-typed envelope strings.

@Serializable data class Ack(val seen: Boolean)
@Serializable data class Token(val value: String, val expiresAt: Long)
 
loadHtml(html = checkoutHtml, baseUrl = "https://app.example.com")
waitFor("#pay")
click("#pay")
awaitMessage(type = "payment-token", into = "token", timeoutMs = 5_000)

Because a workflow is a builder, the reply cannot come back into the block — it lands in a variable, and the typing picks up where the value does: event.decodePayload<Token>("token"). When you want the round trip as a single expression, you want the host API instead, controller.bridge.request<Ack, Token>(...), which posts and correlates the reply for you. The subtlety of why post-then-await does not race — the page can reply before you start waiting — is one of the things part two* has to get right.

A run, on screen

The sample app is a gallery of these workflows, and running one shows the page on top with every step underneath and what it actually did.

The same composeApp runs on iOS and the desktop from the same source; past 720dp wide the list and the runner sit side by side instead of taking turns. vitre-core has no Compose dependency at all — the DSL, the engine, the bridge and the controllers are pure Kotlin Multiplatform, and the Compose layer is a thin VitreWebView on top for people who want it.

What is deliberately out of scope

The browser is not a target, and it cannot be. The whole value proposition rests on the app being the one thing that can answer for the network — relax CORS, serve a fixture, read a first-party cookie jar. In a browser tab you have none of that; you have the same-origin policy the page's own author has, and a general-purpose automation framework there is impractical for reasons that have nothing to do with Vitre. So Vitre is mobile and desktop only, and honest about it.

That is the shape of the library. The next post* is about the part I underestimated by an order of magnitude: what happens when the UI, a workflow, and an agent all want to touch the same WebView at the same time, on a thread none of them are allowed to block.