feat: add --progress-file for machine-readable case progress events
- Dominant language
- Go
- Stars
- 894
- Forks
- 68
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 38
Description
## Problem / motivation
`skill-up run` currently exposes case-level progress only through human-readable terminal output.
CI systems that need to periodically report evaluation progress must parse stdout, which is fragile because:
- the output format is intended for humans rather than machines;
- concurrent cases may interleave output;
- verbose logs may contain unrelated messages;
- wording or formatting changes can break downstream parsers.
`evaluator.ProgressObserver` already receives callbacks when cases start and complete, but these events are only forwarded to the terminal UI.
We need a stable, machine-readable progress stream that CI jobs can consume while an evaluation is still running.
## Proposed solution
Add an optional `--progress-file` flag to `skill-up run`:
```bash
skill-up run --progress-file ./progress.jsonl
```
When specified, skill-up should:
1. Create or truncate the target file when the run starts.
2. Append one JSON object per line as `ProgressObserver` callbacks occur.
3. Make each event immediately visible to readers so CI wrappers can tail or periodically read the file.
4. Preserve the existing terminal progress output.
5. Leave current behavior unchanged when the flag is not specified.
Example events:
```json
{"schema_version":"1","event":"case_started","time":"2026-08-18T10:00:00.002Z","iteration":1,"case_index":1,"case_total":2,"case_id":"case-1","configuration":"with_skill","title":"Basic flow"}
{"schema_version":"1","event":"case_completed","time":"2026-08-18T10:01:10.451Z","iteration":1,"case_index":1,"case_total":2,"case_id":"case-1","configuration":"with_skill","status":"PASS","pass_rate":1.0,"duration_ms":70449}
```
Suggested fields:
| Field | Description |
| --- | --- |
| `schema_version` | Event schema version, initially `"1"` |
| `event` | `case_started` or `case_completed` |
| `time` | Event timestamp in RFC 3339 format |
| `iteration` | Current iteration number |
| `case_index` | Existing observer case index |
| `case_total` | Total number of case tasks in the iteration |
| `case_id` | Case identifier |
| `configuration` | `with_skill` or `without_skill` |
| `title` | Case title, for `case_started` |
| `status` | `PASS`, `FAIL`, `ERROR`, or `SKIP`, for `case_completed` |
| `pass_rate` | Judge pass rate when available |
| `duration_ms` | Case duration, for `case_completed` |
The progress writer should be safe when cases run concurrently. Each JSON line must be written atomically under synchronization so readers never observe interleaved or partial events.
Supporting benchmark and multi-iteration runs requires passing `configuration`, `iteration`, and duration information through the observer path. In benchmark mode, consumers can distinguish duplicate case IDs using `(case_id, configuration)`.
Example CI usage:
```bash
skill-up run --progress-file ./progress.jsonl &
skill_up_pid=$!
while kill -0 "$skill_up_pid" 2>/dev/null; do
completed=$(jq -s '[.[] | select(.event == "case_completed")] | length' progress.jsonl 2>/dev/null || echo 0)
echo "Completed cases: ${completed}"
sleep 10
done
wait "$skill_up_pid"
```
### Error handling
- If the progress file cannot be created or opened, fail at startup with a clear error because the user explicitly requested the output.
- If writing fails after evaluation has started, emit a warning and allow the evaluation to continue with its normal result and exit code.
### Acceptance criteria
- [ ] `skill-up run` accepts `--progress-file `.
- [ ] No progress file is created when the flag is absent.
- [ ] The file is created or truncated once at run startup.
- [ ] Each observer callback appends exactly one newline-terminated JSON event.
- [ ] Events are visible while evaluation is still running.
- [ ] Every line is independently valid JSON.
- [ ] Concurrent case execution cannot produce interleaved or corrupted lines.
- [ ] Events distinguish `with_skill` and `without_skill` executions.
- [ ] Events identify the current iteration.
- [ ] Existing terminal progress output remains unchanged.
- [ ] Tests cover event serialization, concurrent writes, benchmark mode, multiple iterations, and write failures.
- [ ] CLI documentation and changelog are updated.
## Alternatives considered
### Parse stdout
This works without code changes, but stdout is a human-facing interface and is not a stable machine-readable contract.
### Poll report files
Reports are generated after an iteration completes, so they cannot provide case-level progress during execution.
### Use OTLP telemetry
OTLP is appropriate for observability platforms, but requiring a collector is unnecessarily heavy for CI jobs that only need to report completed case counts.
### Add an HTTP callback
A callback would introduce authentication, retry, timeout, and delivery semantics. A local JSONL stream keeps skill-up simple and lets CI wrappers choose how to forward progress.
## Additional context
The existing integration point is `evaluator.ProgressObserver`, currently implemented by `uiProgressObserver` in `internal/cli/run.go`.
The new file observer can be composed with the existing UI observer so both receive the same callbacks.
Contributor guide
Research direction
Start with evaluator.ProgressObserver and its uiProgressObserver implementation in internal/cli/run.go, then trace how benchmark and multi-iteration runs pass case metadata. Add the optional progress-file path while preserving terminal output, and verify JSONL serialization, synchronized concurrent writes, startup and write failures, benchmark and iteration fields, CLI documentation, and the changelog.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100