--- title: "Jev Doesn't Generate. That's the Point: Building a Browser QA Loop on TypeSafe's System One Model" date: 2026-09-19 author: Jonathan Muszkat canonical: https://me.jonymusky.com/blog/jev-system-one-browser-qa --- # Jev Doesn't Generate. That's the Point: Building a Browser QA Loop on TypeSafe's System One Model On September 15 a model appeared on Vercel's AI Gateway that cannot write a sentence. It cannot call a tool, it cannot explain itself, and its maximum output is zero tokens. Three days later Vercel [wrote](https://vercel.com/blog/ai-gateway-jev-model-launch) that it had reached nearly 13% of paid teams in its first 24 hours, twice the share of any previous launch and six times what Fable 5.1 managed on day one. The model is Jev, from a company called TypeSafe, and it is the first of what they call System One models. I spent this week building a browser QA loop on top of it. This is what it is, why I think the excitement is deserved, what I built, and what the numbers look like next to the LLMs you are already paying for. One practical note before the argument: at the time of writing, Vercel is running a launch promotion and Jev is free on AI Gateway through September 25. After that it is USD 0.042 per million input tokens, and output is free because there is no output. Either way it costs about nothing, which is part of the story. ## What Jev is, in one paragraph You give Jev a **state** (a string, a JSON object, up to 32k tokens) and a set of **questions**. Each question is one of three primitives: a **boolean** ("does this claim hold?"), a **choice** ("which of these options?") or a **score** ("where on this ordered scale?"). It answers every question with a typed value and a probability distribution, and nothing else. No prose, no reasoning trace, no JSON to parse. TypeSafe trains it with a method they call Reinforcement Learning for Calibrated Decisions, so the probabilities are optimized against outcomes rather than against a human's preference for a nice answer. A 0.97 is meant to be right about 97% of the time. Through the AI SDK it looks like this: ``` import { experimental_evaluate as evaluate } from 'ai'; const result = await evaluate({ model: 'typesafe-ai/jev', state: { url, title, visibleText }, questions: { signedIn: { type: 'boolean', instructions: 'Is the user signed in, judging from the page?', }, section: { type: 'choice', instructions: 'Which section is the page showing?', criteria: { dashboard: null, login: null, other: null }, }, }, }); // result.answers.signedIn.probability -> 0.9 // result.answers.section.choice -> 'dashboard' // result.answers.section.probabilities -> { dashboard: 1, login: 0, other: 0 } ``` That is the whole API. The `ai` package added `experimental_evaluate` in 7.0.105 for exactly this model class, and the gateway exposes it as an "evaluation model" next to language, embedding and image models. It is a new column in the catalog, not a new row. ## Why this is a bigger deal than a cheap classifier Look at where LLM calls actually sit in a production codebase. A minority generate text a human will read. The majority make a decision code will act on: route this ticket, does this CV match the knockout question, is this message urgent, which of these five candidates is the one the user meant, did this step succeed. For three years the way to get that decision has been to ask a chat model to please answer in JSON, parse the JSON, retry when it does not parse, and treat the answer as binary because the model gives you no honest number for how sure it is. Every one of those calls pays for generation. You pay for the output tokens, you wait for them to stream, and you inherit a failure mode (malformed output) that has nothing to do with the decision. Structured output modes made the parsing more reliable but did not make it faster or cheaper, and logprobs, where available, are not calibrated. Jev removes the generation step entirely. The decision *is* the output. There is nothing to parse, nothing to retry for formatting reasons, and the number that comes back means what it says. The consequence is that decisions become cheap enough to make many of them. Ask five questions about the same state and they run in parallel in one request. Ask a speculative question you might not need. Score every candidate on four dimensions and let the ranking be a weighted sum you can change without another inference. That is a different way of writing software around a model, and it is why TypeSafe's own docs read more like a programming guide than a prompting guide. The model is a primitive. Code owns the workflow. What Jev cannot do is just as important, because the limits are the design. It does not generate, so it cannot write a selector, a summary, an email or a line of code. It cannot plan. It reads text, not images. If your task needs any of that, you still need a language model. The right mental model is not "a smaller LLM" but "the fast, calibrated judgment you put in front of, beside or after the LLM". ## What I built: a QA loop where Playwright drives and Jev judges End-to-end browser testing is a good place to try this, because it is exactly the workload where the current tooling spends a frontier model's turn on every click. Agent-driven QA tools (Expect, browser-use and friends) hand a coding agent a browser and let it plan, click, type and verify. It works, it is slow, and it is expensive in the way that adds up: a 20-step flow is 20 or more agent turns. The split I ended up with is simple and I would defend every line of it: Playwright (code)Jev (model) Navigation, typing, waitingyes Video, trace, screenshotsyes "Does this claim hold on the page?"yes, a boolean question "Which visible control matches this intent?"yes, a choice among the controls code found "What is the next click toward this goal?"yes, a choice among observed actions or DONE / BLOCKED / WAIT Text entry never goes near the model. Code takes a snapshot of the page (URL, title, deduplicated visible text, the aria tree, and every visible clickable control tagged with an index), and Jev only ever chooses among things code already found. It never invents a selector. That is the same shape as the [jev-browser-use](https://github.com/wy-coliney/jev-browser-use) bridge someone published for Codex this week, adapted to plain Playwright so it runs anywhere. On top of the three primitives I built the things a team actually needs: - **A CLI for agents.** A flow is a JSON array of steps. The CLI runs it headless, films it, and prints one JSON result with pass or fail per step, the probability behind each judgment, the path to the video and the path to a final screenshot. Exit code 0 or 1. Credentials are referenced as `${QA_EMAIL}` placeholders and substituted from the CLI's own environment, so an agent writing a flow never sees them. - **A dashboard.** A single HTML file served locally, in English, Spanish or Portuguese. Every run, every test, every Jev decision with its probability, latency and token count, the video inline. - **A skill.** A `SKILL.md` that teaches Claude Code, Codex, Cursor or Grok Build how to write a flow and how to read the result. "Test this branch in the browser" becomes something an agent can do without a human watching. The dashboard after an afternoon of runs. 545 ms mean Jev latency across the five decisions of the last flow. Here is a real run against our product. The flow logs in deterministically, then hands Jev a goal, then asks it to verify: ``` { "goal": "Open the account or workspace settings screen. Do not change any setting.", "maxSteps": 8, "denyNames": ["cerrar sesi", "logout", "eliminar", "delete", "guardar", "save"] }, { "expect": "A settings or configuration screen is displayed." } ``` ``` [8/9] goal: Open the account or workspace settings screen. Do not change any setting. step 1: Click button "Menú de usuario: Olivia Owner" p=0.76 step 2: Click link "Mi cuenta" (href=/settings) p=0.99 step 3: WAIT p=0.38 step 4: DONE p=1.00 passed (3245 ms) [9/9] expect: A settings or configuration screen is displayed. passed p=0.99 (443 ms) ``` Four decisions, 3.2 seconds including the page loads, and the third one is my favourite: the page was mid-transition after the click, and instead of clicking something else Jev chose WAIT with a low probability, which is exactly the right thing for a model to say when the state is ambiguous. In the login flow, asked for "the button that submits the login form", it picked "Iniciar sesión" at 0.97 from a Spanish page it had never seen. The intent was in English. It did not matter. The same run in the dashboard. Every decision is inspectable after the fact, with the video next to it. ## The numbers I measured Jev myself. Twenty sequential calls through AI Gateway from Buenos Aires, each with a boolean and a choice question over a realistic page state of about 400 tokens: Jev via AI GatewayValue p50 latency (round trip)388 ms p90 latency471 ms min / max330 ms / 883 ms Input tokens per call395 (judgments), ~1,500 (goal steps with the control list) Cost per judgmentUSD 0.0000166. A thousand assertions for under two cents. The comparison below is the part to read carefully. I did **not** run the other models: I took list prices from each provider's pricing page and latencies from Artificial Analysis, and I computed what the same judgment would cost if you asked a language model for it as a small JSON object: 750 input tokens (the state plus the question) and 25 output tokens. Time to first token is the published figure for the non-reasoning configuration where one exists; total time adds those 25 tokens at the published output speed. Treat the latency column as an estimate and the cost column as arithmetic. ModelPrice in / out (USD per 1M)Cost per judgmentvs JevEstimated latency **Jev (measured)**0.042 / 00.00003151x**0.39 s measured** GPT-5 nano0.05 / 0.400.00004751.5x~0.5 to 1 s without reasoning (76 s with reasoning on high, per Artificial Analysis) GPT-5 mini0.25 / 2.000.000247.6x~1 s without reasoning Gemini 3.5 Flash-Lite0.30 / 2.500.000299.2x~0.5 s without thinking (9.4 s with thinking) Claude Haiku 4.51.00 / 5.000.00087528x~1.0 s (0.69 s TTFT + 25 tokens at 83 tok/s) Gemini 3.5 Flash1.50 / 9.000.0013543x~1 s without thinking (15 s with) Claude Sonnet 52.00 / 10.000.0017556x~1.5 to 2 s Claude Fable 5.110.00 / 50.000.00875278xseveral seconds Three things stand out. First, the cheapest LLMs are within a small multiple of Jev on price, so cost alone is not the argument at the nano tier; latency and calibration are. Second, the models people actually reach for when they want a judgment they can trust (Haiku, Flash, Sonnet) are 30 to 60 times the cost and two to five times the latency, and they still do not give you a probability. Third, TypeSafe's own headline number, "194x faster and 445x cheaper", is against reasoning-style workflows, and once you look at the reasoning latencies in that table (76 seconds for nano on high) it stops sounding like marketing. For the QA loop the comparison that matters is per flow: the settings run above made five decisions for USD 0.0003. Run it after every commit on every branch and the bill is still zero. ## What I learned in a week - **Write claims the way a QA engineer writes expected results.** "A list of job positions is displayed, or an empty state inviting to create the first one" works. "The page is correct" does not. When a claim is subtle, give it explicit true and false criteria; the model uses them. - **A probability near your threshold is feedback on the claim, not on the threshold.** Sharpen the question. Do not lower the bar. - **Deny lists are not optional in an action loop.** Jev will happily choose "Cerrar sesión" if it is the only control that looks like progress. The loop only offers unique, permitted controls, and destructive names are never offered. - **Text, not pixels.** Jev never sees the screenshot. Visual regressions still need a vision model or a pixel diff. The screenshot in the report is for the human. - **Calibration is per population, not per answer.** The docs say so and it shows: a 0.76 on "click the user menu" was the right click, and a 0.38 on WAIT was the right hesitation. Read the distribution, not just the argmax. - **It composes with the agent tools you already use.** The natural home for this is not replacing Expect or a coding agent, it is being the mechanical layer they call so they stop spending a turn per click. That is precisely what the Codex bridge does, and what the CLI plus skill here does for everything else. ## Try it The project is open source at [github.com/jonymusky/jev-browser-qa](https://github.com/jonymusky/jev-browser-qa). It is small on purpose: three primitives, a CLI, a dashboard, a skill, and a README that says what Jev will not do. ``` git clone https://github.com/jonymusky/jev-browser-qa && cd jev-browser-qa pnpm install && pnpm exec playwright install chromium cp .env.example .env # AI_GATEWAY_API_KEY, BASE_URL, QA_EMAIL, QA_PASSWORD pnpm jev:smoke # one call, no browser pnpm -s qa run flows/login.json pnpm dashboard # http://localhost:4310 ``` And for your agent: `npx skills add https://github.com/jonymusky/jev-browser-qa --skill jev-qa`. I have been running it from Claude Code and from [Grok Build](https://docs.x.ai/build/overview), xAI's terminal agent, which reads Agent Skills and `AGENTS.md` with no configuration: the same prompt, "test this branch with jev-qa and report the verdict with the video path", works in both, and neither agent ever sees a credential because flows carry `${QA_EMAIL}` placeholders that the CLI resolves. There is a short [Grok Build walkthrough](https://github.com/jonymusky/jev-browser-qa/blob/main/docs/agents/grok-build.md) in the repo, including the headless `grok -p` form for CI. If you want to try it without installing anything, I published a Grok bot with the skill preloaded: [QA Engineer](https://x.ai/bot/IoOXnqvhpn4i_C1TNq9P_). Give it the URL of your app and the flow you want verified, and it comes back with the verdict, the video and the screenshot. I have been careful in this post to separate what I measured from what I inferred, because the claims around Jev are large and the model is four days old. But the design is not a trick. A model that returns a calibrated decision instead of a paragraph, in 400 milliseconds, for a fraction of a cent, changes which parts of your system it is reasonable to put a model in. Until now the answer was "the parts that can afford it". As of this week the answer is closer to "wherever you would otherwise write an if statement and feel bad about it".