feat(cli): add offline result compare command
- Dominant language
- Go
- Stars
- 894
- Forks
- 68
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 38
Description
## Background
Now skill-up can run evals, generate reports, and create baseline benchmark data. But I think it still misses one offline command to compare two finished runs.
When we change a Skill, review a PR, or upgrade model, we often want to know:
- which cases are fixed
- which cases are regressed
- how pass rate changes
- how tokens and duration change
- whether engine or model is different
- whether this can be used as a CI gate
## Proposal
Add a new command:
```bash
skill-up compare
```
It should compare two existing result/report JSON files. It should not run eval again, and should not call any agent or judge.
## Scope
The command can:
- read JSON files compatible with current report.Input/result.json format
- use primary case result semantics, so benchmark without_skill entries are not counted twice
- show run level delta:
- pass rate
- case counts
- total/input/output tokens
- duration
- show case level transition:
- fixed
- regressed
- unchanged
- added
- removed
- show metadata difference, like skill name, engine, model, schema version and time
- support default text output
- support --format json for CI or other tools
- support optional gates, for example fail on regression or too much token increase
## Non-goals
- Do not rerun evals
- Do not call agent or judge
- Do not add dashboard/cloud/snapshot feature in this issue
- First version does not need markdown output
## Acceptance criteria
- `skill-up compare old.json new.json` prints a readable summary
- `--format json` prints stable machine readable output
- benchmark result is not double counted
- fixed/regressed/added/removed cases are classified correctly
- configured gate failure returns non-zero exit code
- invalid input gives clear error message
Contributor guide
Research direction
Start by locating the CLI command registration and the current report.Input/result.json format. Check how existing results represent primary and benchmark cases before defining the comparison output, transitions, and gate options. Done means readable text and stable JSON output, correct case classification, clear invalid-input errors, and non-zero status for failed gates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100