Trials and review

Validation and smoke say a package runs. Only trials say it is worth running. Review the package as a skeptic before you call it a draft, then trial it against the raw model.

Review#

Review as a skeptic who has to run this package on paying work tomorrow. Treat the author's own description of the package as untrusted. Cite the file and line for every finding, rank findings by how much they would hurt a real run, and mark lines that change no behavior for deletion.

Does it make the model prove the work?#

• Is there an executable gate, invoked by its skill scripts/ path or declared in tools when the image installs it on PATH, whose exit code is the stated definition of done?
• Is the gate manifest-driven, with numbered runs, so a failure cannot be deleted or overwritten?
• Do the instructions name the specific cheats the gate invites?
• After the gate passes, must the model open the evidence and judge it?

Does it encode expertise a capable model lacks?#

• Is there a written failure list, and does each failure map to a rule, a skill, a gate check, or a template?
• Are there field rules that only someone who has done this work would know?
• Is taste encoded as a design artifact, a rubric with level descriptions, anchors, thresholds, and automatic failures, and a named blacklist?
• Is there any hedging, generic advice, or restated manifest? Mark it for deletion.

Is it coherent?#

• One entry point that routes to skills split by judging axis or phase.
• No skill references a tool, worker, MCP server, or interaction the package lacks.
• Scope classes (new, upgrade, narrow edit) with their own bars, the user's scope winning over defaults, and explicit stop conditions.

Is it sound to run?#

• The newest spec your installed wb supports, a version bumped for this change, and wb validate, wb view, and wb smoke passing.
• Every executable the instructions mention is declared in tools and present in the runtime. Package scripts are invoked by path instead.
• runner_config enforces what the instructions promise.
• Images pin versions, tolerate an arbitrary user and a read-only root, and comment why each dependency exists.
• No credential values, secret files, or .env contents anywhere.
• Paths are package-relative and workspace-relative; no ...

Is there evidence?#

• Has it been trialed on representative tasks against the raw model, with the worked count and cost per working result reported?
• Is the model choice backed by that evidence, or stated as unevaluated?
• Has it been exercised end to end, not only validated and smoked?

A review that finds nothing on a first pass did not look.

Trials#

Before spending anything#

Trials make model and sandbox requests that cost money. Write down the plan (tasks, arms, attempts, model, runtime) and an estimated cost, agree a budget with whoever pays, and cap every attempt's spend or duration.

Design#

• Tasks: three to eight representative tasks from the class of work, written the way a real user would ask. Include at least one new build and one narrow edit if the package claims both.
• Arms: the package, and the same model with no package. Same runner, same runtime, same task text.
• Attempts: at least three per task per arm. Single runs mislead.
• Grading: blind to the arm. Prefer a separate grader, such as a second Workbench or a script, with hard pass or fail checks derived from the failure list: it launches, the first run is honest, data persists, delete exists, the gate's own checks pass. Add rubric scores for taste where it matters.

Register the package once with wb add ./.workbenches/<name> --as <alias>. A local registration is live: every new run reads the current files, so edit and rerun without re-adding.

Accounting#

• Worked: an attempt passes every hard check.
• Cost per working result: total spend of the arm divided by worked attempts. A package that costs five times more per run but works five times as often costs the same per working result and is strictly better.
• Take each run's spend from usage_total.cost_usd in its terminal wb wait --json result. Do not sum the per-wait usage deltas across runs or waits.
• Separate infrastructure failures (network, quota, runner faults) from work failures. Fix or rerun them; do not count them against the package.

Iterating#

For every failed attempt, read the transcript until you can name the cause, then classify it:

• a missing or wrong rule in the instructions,
• a procedure or reference that belongs in a skill,
• a gate gap that let bad work pass or good work fail,
• a missing template the model rebuilt badly,
• an undeclared tool, binding, or runtime requirement,
• a model that cannot do the work, which is the only finding that justifies changing the model.

Make the smallest general fix that would prevent the class of failure, never one that encodes the task, its wording, or its answer. Increment the version, validate, smoke, and rerun the affected tasks. Keep a short ledger of version, change, and result so each rule in the package traces to evidence.

Report#

Per arm: attempts, worked, total spend, cost per working result, and the top failure causes. Then the changes made and what still fails.

Improve from real runs#

wb create --from <session> improves a Workbench from a session's bounded, normalized evidence. Feedback is optional:

wb create --from wb_... --feedback "The migration path missed our rollback convention"

The creator edits the source package, not the immutable package the session loaded. Run evidence is treated as untrusted data. Changes apply only to future runs; every session resumes from the package bytes it captured at start.