alibaba / alibaba/skill-up

feat(cli): add offline result compare command

Open
#175 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
894
Forks
68
Avg merge
1d 15h
Merged PRs (30d)
38

Description

## Background

Now skill-up can run evals, generate reports, and create baseline benchmark data. But I think it still misses one offline command to compare two finished runs.

When we change a Skill, review a PR, or upgrade model, we often want to know:

- which cases are fixed
- which cases are regressed
- how pass rate changes
- how tokens and duration change
- whether engine or model is different
- whether this can be used as a CI gate

## Proposal

Add a new command:

```bash
skill-up compare
```

It should compare two existing result/report JSON files. It should not run eval again, and should not call any agent or judge.

## Scope

The command can:

- read JSON files compatible with current report.Input/result.json format
- use primary case result semantics, so benchmark without_skill entries are not counted twice
- show run level delta:
- pass rate
- case counts
- total/input/output tokens
- duration
- show case level transition:
- fixed
- regressed
- unchanged
- added
- removed
- show metadata difference, like skill name, engine, model, schema version and time
- support default text output
- support --format json for CI or other tools
- support optional gates, for example fail on regression or too much token increase

## Non-goals

- Do not rerun evals
- Do not call agent or judge
- Do not add dashboard/cloud/snapshot feature in this issue
- First version does not need markdown output

## Acceptance criteria

- `skill-up compare old.json new.json` prints a readable summary
- `--format json` prints stable machine readable output
- benchmark result is not double counted
- fixed/regressed/added/removed cases are classified correctly
- configured gate failure returns non-zero exit code
- invalid input gives clear error message

Contributor guide

Open the contributing guide

Research direction

Start by locating the CLI command registration and the current report.Input/result.json format. Check how existing results represent primary and benchmark cases before defining the comparison output, transitions, and gate options. Done means readable text and stable JSON output, correct case classification, clear invalid-input errors, and non-zero status for failed gates.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.