Building Vault with three Workbenches

The setup

Vault is a secrets manager for small teams. It has workspaces with roles, projects with development, staging and production environments, versioned secrets with rollback, scoped service tokens, an audit log, and a CLI whose vault run -- <cmd> injects secrets as environment variables.

Three Workbenches did the building, each on GPT 6.1 Sol through OpenCode, running in cloud sandboxes:

Claude Code orchestrated the whole thing through the Workbenches MCP server. It wrote each task, started the run, answered the run's questions, reviewed the pull request, and handed the result to the next Workbench. Each feature went the same way: backend, then UI, then browser tests, then a fix run if the tests found bugs. Runs went one at a time, and each pull request stacked on the last.

A person stayed in the loop for decisions: what to build next, whether to change a Workbench, and clicking through the app after each merge.

What it looks like

These screenshots come from a throwaway copy of Vault filled with example data: a team called Acme, three projects, and three people. Maya is the owner, Priya an admin, and Sam a member.

And the part developers use every day. Sam links a directory to an environment once, then runs his app through vault run:

Real output from the demo. The vault run line stands in for the command we ran, which printed two of the injected variables.

What it cost, feature by feature

Each row runs from the first backend run to the last passing test. Costs are model usage only.

PRs #13, #14, #25 and #37 were small changes Claude Code made directly: two UI fixes, removing generated report files, the CI workflow, and the hosting setup.

Two things stand out. First, a feature after the MVP cost $1 to $6 and took 20 minutes to 2 hours, including its browser tests. Second, almost half of everything went into one run: the first browser-test run, at $13.02 and 3 hours 37 minutes. That run is where the Workbench improvements started.

Nineteen runs failed before their agent started, mostly on the first day, while we got the three Workbenches running in cloud sandboxes for the first time. Those failures cost nothing in model usage. The one stalled run that did cost something cost $0.11.

Improving the Workbenches while building with them

Every run leaves evidence: a transcript, a gate result, a pull request. When a run went wrong, we asked whether the Workbench let it go wrong, and fixed the package rather than the one repo. A Workbench version is immutable, so each fix became a new version, and the next run proved whether it worked. Over four days the three packages went through 34 published versions.

1. e2e 0.1.31hono 0.1.26

   Runs that hang silently

   The first browser-test run hit a permission prompt nobody saw and sat idle for 2.5 hours. The next one stalled after Playwright finished, because the test process never exited.

   The service tool gained a hard timeout. The gate gained one too, plus the ability to run named spec files. The runner allowed the sandbox paths it kept asking about. The instructions gained a run budget: iterate on single files, run the full suite once, at most twice.

   Result: No browser-test run has hung since.

2. e2e 0.1.32hono 0.1.27

   A 45-minute test suite

   The suite slept 62 seconds before every test to stay under the API's auth rate limit, and ran on one worker. The instructions already said "run in parallel". The agent had a reason not to, and nothing measured speed, so a slow suite still counted as done.

   We made speed part of the gate. It fails on fixed sleeps in tests, or on an average of more than 20 seconds per test across a full run. We added a parallel Playwright config and a sign-in-once setup as templates. Raising a rate limit through the app's own config in a throwaway test environment became explicitly allowed. And when the app is what makes tests slow, the Workbench writes a handoff instead of working around it.

   Result: Covered in the experiment below.

3. hono 0.1.29e2e 0.1.34

   Rules that were too narrow

   The backend Workbench hit 429s in its own tests and added retries, because its rate-limit rule only covered "code you touch". OpenCode also cut off the full browser suite at its default 120-second command limit.

   The backend rule now treats any 429 in its own tests as a sign the limit should be configurable, and bans retries and sleeps around a rate limiter. The e2e instructions now run long gate commands in the background or with a longer timeout.

4. all three

   Files nobody asked for

   Browser-test runs committed their reports to the repo as Markdown, justified by an allowance for "test-only documentation".

   All three Workbenches now create no new Markdown files unless the task names one. Reports go in the pull request and the final message.

5. hono 0.1.30e2e 0.1.37

   Pausing to ask for test config

   Runs stopped to ask for values like a test email sender or a reset-link lifetime, which aren't secrets.

   When .env is missing, a Workbench now builds its throwaway test environment itself. Plain settings come from .env.example, and secrets are fresh random values, never printed or committed. The example mail sender is allowed only while mail goes to the sandbox's Mailpit.

6. svelte 0.1.29

   A field with no job

   Users had to pick a workspace slug, but URLs used the internal ID. We noticed by using the app; no test asked about it.

   The frontend Workbench now has a rule: when users choose a readable identifier, URLs use it, and a field the user must fill has a visible job.

   Result: The next run moved URLs to slugs and made building a URL from an ID a type error.

7. e2e 0.1.35

   A gate that was too strict

   A targeted rerun of three screenshot-heavy files failed the speed check at 23.1 seconds a test, though the full suite passed at 15.9.

   The average now fails only full-suite runs. Targeted runs get a warning, and still fail on sleeps.

The experiment: does a Workbench fix itself?

To test the speed changes, we built a small feature, project descriptions, without mentioning speed, sleeps or rate limits in any task. The question was whether the new gate alone would get the existing slow suite fixed.

It did. The browser-test run found the 62-second sleeps, removed them, moved to 4 workers and API sign-in once per role, and hit 429s on the older tests. As instructed, it didn't work around them. It wrote a handoff:

Handoff: application changes needed (condensed from PR #17) apps/api/src/auth.ts:85 hardcodes 100 requests per 60 seconds, and Better Auth limits sign-up and sign-in to 3 per 10 seconds. Older specs that sign up through the UI hit 429s: 13 of 29 failed in setup. Make these limits configurable, keeping production values as defaults.

The Workbenches don't know about each other, by design. Each package stands alone and never names another. So the orchestrator relays a handoff like this one: it passed the text to the backend Workbench, which made six rate limits configurable without changing production behavior. Then it ran the full suite once more.

It has held since. By the end, the suite had 82 tests and ran in 4 minutes 42 seconds. Browser-test runs that wrote new coverage cost $1.10 to $2.94 each, against $13.02 for the first one.

Under the hood

A demo can look finished while the code underneath is hard to change. So we looked at the repository the way we'd look at a contractor's: how it's laid out, what adding a feature costs, and what would slow the next person down.

Structure

The shape is conventional, which is a compliment. Every request and response is defined once in shared/contracts, and the API, dashboard and CLI all import it, so a renamed field fails typecheck everywhere at once. Identity, sessions and organizations come from Better Auth instead of hand-rolled code. Secrets use envelope encryption: a master keyring wraps one data key per project. The dashboard is a static build with no server of its own, so the API is the only thing that ever holds a credential.

Extensibility

Each feature landed in the same few places, so the next one is easy to predict. Project descriptions is the smallest clean example. The backend commit changed about 22 lines of application code: 4 in the schema, a 2-line migration, 8 in the projects route and 8 in the shared contract. It came with a 163-line integration test. The UI side was three form components, a two-line API client change, and four lines on the overview page. Bigger features, like per-environment access, followed the same path through contract, route, UI and browser tests, just with more of each.

Most of that list can be fixed in the Workbenches, which makes it the next round of Workbench changes. A formatter config and a real linter in the backend package would end the dense lines. A "no build notes in the UI" rule in the frontend package would catch the leftover helper text. As with the speed check, a gate measures these things better than prose does.

How good is it?

Vault is running for our team on one of our own machines, reachable only over Tailscale, with real email, nightly backups and master-key rotation. Every pull request runs typecheck, lint, the API integration suite and the full browser suite in CI.

The weakest result came from the orchestrator, not a Workbench. The first dashboard task told the frontend Workbench not to touch the shared contracts. Those contracts had no sign-up or workspace schemas, so the first dashboard had no way to sign up. One backend run and one frontend run, $2.13 together, filled the gap. The lesson: a Workbench follows its task's boundaries exactly, so the task has to draw them in the right place.

Still missing: two-factor login, a dark theme, an installable CLI, and choosing a project by name instead of by ID when linking a directory.

What we'd tell someone doing this

Fix the package, not the repo. When a run goes wrong, the cheapest fix is often one rule or one gate check in the Workbench. Every later run, in every repo, gets it for free.

Make the gate measure what you care about. Telling the agent to run tests in parallel didn't change anything. Failing the gate on fixed sleeps did, on the first try, without being asked.

Keep Workbenches separate and relay between them. The handoff format let the browser-test Workbench ask for an API change it isn't allowed to make. The orchestrator carried it over in one run.

Keep tasks narrow. The cheapest runs were the narrowest: three named test files and two attempts for $0.30, or a two-file fix for $0.24.