Benchmarks
Same model, same prompts, same spend: 36 working results with Workbenches, 20 without.
| Arm | Working | Model spend | Per working result |
|---|---|---|---|
| Plain | 20 of 40 | $10.59 | $0.53 |
| Workbench | 36 of 40 | $10.64 | $0.30 |
Both arms run the same agent, OpenCode 1.18, on the same model; the only difference is the Workbench. Eight tasks, two arms, five attempts per task and arm, one model (openai/gpt-5.6-terra through OpenRouter). Both arms get the identical request in the identical starting project. Dollars per working result is every attempt's model cost, failures included, divided by the attempts that worked.
Ship a Godot game
Meteor Dash
Make a small Godot game called Meteor Dash. You steer a little ship left and right and dodge falling meteors, and it gets faster the longer you last. Show the score. I want to put it on my site, so leave me a web build I can upload.
- Workbench 5 of 5 $0.16 per working result
- Plain 4 of 5 $0.28 per working result
Lantern Keep
Build a one-screen platformer in Godot called Lantern Keep. Jump between platforms, grab the three lanterns, reach the door. Arrow keys and space. Leave me a browser build I can drop on any static host.
- Workbench 4 of 5 $0.22 per working result
- Plain 1 of 5 $1.12 per working result
Workbench failure: The player spawns at the edge of its platform and drops out of the world on the first Left tap, with no respawn.
Build on Lux
Team announcements
Add team announcements to this app. Everyone on the team should see them show up live, and only whoever posted one can edit or delete it. Before you call it done: the app should start, and `bun run check` should pass.
- Workbench 4 of 5 $0.64 per working result
- Plain 2 of 5 $1.16 per working result
Workbench failure: The route called .single() on an insert, which the project's SDK version does not have, so every post threw.
Invite rate limit
People are spamming invites. Cap it at 5 an hour per team. Before you call it done: the app should start, and `bun run check` should pass.
- Workbench 5 of 5 $0.32 per working result
- Plain 3 of 5 $0.56 per working result
Zero-downtime migration
Widen a column on a live table
events.user_id is an int and we're about to run out. Move it to bigint. Table's huge and we can't take the app down.
- Workbench 3 of 5 $0.43 per working result
- Plain 0 of 5 never worked
Workbench failure: The backfill re-selected the first row not yet copied on every batch, re-scanning the filled prefix each time; on ten million rows it did not finish inside the five-minute migration budget.
Rename a column under a rolling deploy
Rename users.fullname to display_name. The old app version keeps running while the new one rolls out.
- Workbench 5 of 5 $0.25 per working result
- Plain 4 of 5 $0.22 per working result
Make it fast
Slow landing page
Our landing page feels slow on phones. Fix it, keep it looking the same.
- Workbench 5 of 5 $0.22 per working result
- Plain 1 of 5 $0.86 per working result
Search that degrades with data
Search gets slower every month as we add products. Fix it before it falls over.
- Workbench 5 of 5 $0.22 per working result
- Plain 5 of 5 $0.21 per working result
Where plain was cheaper
- Rename a column under a rolling deploy: Plain $0.22 per working result, Workbench $0.25.
- Search that degrades with data: Plain $0.21 per working result, Workbench $0.22.
What this does not show
- We wrote the tasks, and chose domains where an expert's judgment changes the outcome. It says nothing about tasks where expertise does not matter.
- A Workbench packages the environment as well as the instructions. The benchmark measures the whole difference and does not separate the two.
- Two tasks run on Lux, our own database. The Lux Workbench carries its exact syntax; a general-purpose database would not have that asymmetry.
- Some Workbench tools measure the same properties the gates check. They hold no expected values and the plain agent is free to write the same checks, but the overlap is real.