[P0] Add paired with-skill/without-skill evaluation and comparison reporting
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 324
Description
Use cases, pain points, and background
The value of a skill is measured by how agent performance changes relative to a matched no-skill baseline. Gym currently supports selecting a skill directory and recording skills_ref, but users must launch separate runs and join the results themselves. That makes it easy for task sets, repeat IDs, seeds, or other configuration to drift, and there is no standard paired verdict or skill-usage report.
Description:
Add a first-class paired evaluation workflow that runs a treatment with a selected skill set and a matched baseline without skills, then produces a joined comparison report.
The workflow should:
- materialize the task set once;
- hold the agent, model, task inputs, repeats, seeds, and runtime configuration constant;
- vary only the skill treatment;
- link baseline and treatment rollouts by task and repeat;
- report task-level and aggregate deltas;
- produce a configurable Pass / Neutral / Fail verdict; and
- report how often each skill was observed as loaded or activated, using explicit agent telemetry rather than inference from response text.
Design:
Extend the existing run-level skills.path and skills_ref model rather than creating skill-specific dataset fields. Persist a pair identifier and treatment metadata in both runs. Reuse existing aggregation and profiling infrastructure where possible.
Define the verdict policy explicitly, including score direction, minimum paired coverage, tie/neutral threshold, and behavior for missing or failed rollouts. Skill-use telemetry should have a small adapter contract for supported agents and report unknown when the runtime cannot provide reliable evidence.
This complements #1235's goal of evaluating agents with and without skills; this issue adds reproducible orchestration and standard reporting.
Out of scope:
- Automatically editing or optimizing skill content.
- Treating an unpaired aggregate average as a paired result.
- Inferring skill activation solely from the final response.
- Changing benchmark-specific reward semantics.
Acceptance Criteria:
- One command or API request launches a matched no-skill baseline and skill treatment.
- Both variants use the same materialized tasks, repeats, seeds, model, agent, and runtime configuration.
- Every rollout records a pair ID, treatment, and skill provenance.
- Results are joined by task and repeat, with missing pairs clearly reported.
- The report includes baseline score, treatment score, delta, paired coverage, and Pass / Neutral / Fail verdict.
- Verdict thresholds and score direction are configurable and recorded in the report.
- Supported agents emit reliable loaded/activated skill telemetry, summarized per skill and overall.
- Unsupported telemetry is reported as
unknown, not guessed. - Partial failures do not silently bias the comparison.
- Tests cover positive, neutral, negative, missing-pair, retry, and telemetry-unavailable cases.
- Documentation includes a minimal paired evaluation example.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing run-level skills.path and skills_ref model, then review the aggregation and profiling infrastructure mentioned in the issue and the related goal in #1235. Define the paired orchestration, report, verdict policy, and telemetry adapter around those existing pieces. Done means matched baseline and treatment runs, paired deltas and coverage, explicit verdicts, failure handling, telemetry status, tests for the listed cases, and a minimal documentation example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100