dwjohnston / dwjohnston/thunderjar

Add iterations to a test run

Open
#6 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Run each (task, model, harness, promptSet) permutation N times instead of once, to surface agent non-determinism rather than treating a single run as ground truth. Default `iterations: 1` (today's behaviour, unchanged).

## Motivation

Agent behaviour is stochastic. A single run per permutation can't distinguish "this prompt set reliably produces X" from "it got lucky this run." Iterations turn each permutation into a distribution.

## Reporting — variance is the signal, generalized over `MeasurementResult` (decided)

Per `docs/terminology.md`, a measurement does NOT return a pass/fail verdict — it returns `MeasurementResult`: `{ outcome: 'measured', data: T }` (T can be a boolean, a count, an object, anything a measurer chooses to return — see `src/measurements.ts`), `{ outcome: 'skipped' }`, or `{ outcome: 'erroredWhileMeasuring', error }`. There is no runner-level "PASS/FAIL" concept, and no short-circuiting — `src/runner.ts` currently runs every measurement for every run regardless of earlier results. Iteration aggregation must respect this: it cannot assume `data` is boolean or that there's a verdict to average.

Aggregation across N iterations, per measurement:
- Tally `outcome` itself across iterations (`measured` vs `skipped` vs `erroredWhileMeasuring`) — a measurement that errors on 2/5 iterations is itself a signal.
- Within `measured` outcomes, group by structural equality of `data` and show counts — e.g. `3× {ok:true, matches:0}, 2× {ok:false, matches:2}` for a grep measurer. For built-in measurers that happen to shape `data` around an `ok` boolean, this naturally reads like a pass rate, but the aggregation itself must stay generic since custom measurers (`registerMeasurer`) can return arbitrary `TValue` (e.g. a token count) where "grouping by equality" is the wrong summary and something like mean/range is more useful.

Compare view sketch (n=5), for a measurer whose `data` happens to carry `ok`:
```
run 1 run 2
skills/sonnet (n=5) skills/opus (n=5)
────────────────────────────────────────────────────────────────────
bun typecheck 5/5 ok 5/5 ok
grep: 'as any' 3/5 ok:false 5/5 ok:true
2/5 ok:true
────────────────────────────────────────────────────────────────────
Duration (avg) 4m 12s 6m 02s
```
Numeric-valued custom measurers would render differently (e.g. mean ± range) — exact display format is still open, but must not hard-code a pass/fail collapse.

## Scope to nail down

- **Storage**: each iteration is a full independent run and gets its own result JSON (same shape as today), just N of them per permutation. Aggregation happens at `compare` time by reading all N files for a permutation — keeps the runner simple.
- **Config surface**: settable both ways —
- `registerExperiment({ ..., iterations: 5 })` sets the default for that experiment (defaults to 1 if omitted).
- `thunderjar run --iterations ` overrides the config default for that invocation, so you can do a one-off deeper run without editing config.
- **Result naming**: current filename scheme is `------.json` (per run directory) — needs an iteration index added so N runs of the same permutation don't collide.
- **Generic aggregation function**: needs a documented contract for how arbitrary `data: T` gets summarized — equality-grouping as the default, with room for a measurer to opt into a different summary strategy (e.g. numeric stats) later.
- **Duration reporting**: average across iterations, per the sketch above — confirm that's the right aggregate (vs. min/max/total).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/runner.ts, src/measurements.ts, and docs/terminology.md, then trace registerExperiment, the thunderjar run command, and compare. Confirm how independent result JSON files are produced and read before defining iteration naming and generic aggregation. Done means iterations work from both config and CLI overrides, preserve MeasurementResult outcomes, and compare reports aggregated results and average duration without assuming boolean data.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.