dwjohnston / dwjohnston/thunderjar
Add iterations to a test run
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Run each (task, model, harness, promptSet) permutation N times instead of once, to surface agent non-determinism rather than treating a single run as ground truth. Default `iterations: 1` (today's behaviour, unchanged).
## Motivation
Agent behaviour is stochastic. A single run per permutation can't distinguish "this prompt set reliably produces X" from "it got lucky this run." Iterations turn each permutation into a distribution.
## Reporting — variance is the signal, generalized over `MeasurementResult` (decided)
Per `docs/terminology.md`, a measurement does NOT return a pass/fail verdict — it returns `MeasurementResult`: `{ outcome: 'measured', data: T }` (T can be a boolean, a count, an object, anything a measurer chooses to return — see `src/measurements.ts`), `{ outcome: 'skipped' }`, or `{ outcome: 'erroredWhileMeasuring', error }`. There is no runner-level "PASS/FAIL" concept, and no short-circuiting — `src/runner.ts` currently runs every measurement for every run regardless of earlier results. Iteration aggregation must respect this: it cannot assume `data` is boolean or that there's a verdict to average.
Aggregation across N iterations, per measurement:
- Tally `outcome` itself across iterations (`measured` vs `skipped` vs `erroredWhileMeasuring`) — a measurement that errors on 2/5 iterations is itself a signal.
- Within `measured` outcomes, group by structural equality of `data` and show counts — e.g. `3× {ok:true, matches:0}, 2× {ok:false, matches:2}` for a grep measurer. For built-in measurers that happen to shape `data` around an `ok` boolean, this naturally reads like a pass rate, but the aggregation itself must stay generic since custom measurers (`registerMeasurer`) can return arbitrary `TValue` (e.g. a token count) where "grouping by equality" is the wrong summary and something like mean/range is more useful.
Compare view sketch (n=5), for a measurer whose `data` happens to carry `ok`:
```
run 1 run 2
skills/sonnet (n=5) skills/opus (n=5)
────────────────────────────────────────────────────────────────────
bun typecheck 5/5 ok 5/5 ok
grep: 'as any' 3/5 ok:false 5/5 ok:true
2/5 ok:true
────────────────────────────────────────────────────────────────────
Duration (avg) 4m 12s 6m 02s
```
Numeric-valued custom measurers would render differently (e.g. mean ± range) — exact display format is still open, but must not hard-code a pass/fail collapse.
## Scope to nail down
- **Storage**: each iteration is a full independent run and gets its own result JSON (same shape as today), just N of them per permutation. Aggregation happens at `compare` time by reading all N files for a permutation — keeps the runner simple.
- **Config surface**: settable both ways —
- `registerExperiment({ ..., iterations: 5 })` sets the default for that experiment (defaults to 1 if omitted).
- `thunderjar run --iterations ` overrides the config default for that invocation, so you can do a one-off deeper run without editing config.
- **Result naming**: current filename scheme is `------.json` (per run directory) — needs an iteration index added so N runs of the same permutation don't collide.
- **Generic aggregation function**: needs a documented contract for how arbitrary `data: T` gets summarized — equality-grouping as the default, with room for a measurer to opt into a different summary strategy (e.g. numeric stats) later.
- **Duration reporting**: average across iterations, per the sketch above — confirm that's the right aggregate (vs. min/max/total).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with src/runner.ts, src/measurements.ts, and docs/terminology.md, then trace registerExperiment, the thunderjar run command, and compare. Confirm how independent result JSON files are produced and read before defining iteration naming and generic aggregation. Done means iterations work from both config and CLI overrides, preserve MeasurementResult outcomes, and compare reports aggregated results and average duration without assuming boolean data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100