apache / apache/magpie

Add a multi-SKIP case so the gfi-sweep summary-only rule can actually fail

Open Beginner friendly
#1,001 0 comments 0 reactions 0 assignees View on GitHub
good first issue
Dominant language
Python
Stars
92
Forks
92
Avg merge
1d 12h
Merged PRs (30d)
104

Description

## Summary

In the `good-first-issue-sweep` Step 3 eval suite, the field that checks SKIP issues are shown
as a count summary is `true` in all four cases, three of them vacuously. Add a case with
several SKIP issues so the rule has something to fail on.

## Background

`skip_count_shown_as_summary_only` encodes a real presentation rule: SKIP issues must appear as
a count line, not as a list of titles and reasons. But of the four existing cases, only
`case-2-mixed` has any SKIP issues at all — and it has one. The output-spec says that when
there are no SKIP issues the field is `true` "vacuously satisfied", so the other three expect
`true` for free.

That means a model that always answers `true` passes the whole suite on this field, including
one that helpfully lists every skipped issue with its reason. The project's own guidance in
`docs/education/eval-driven-development.md` names this pattern under "All your cases expect the
same value".

The fix is a new case whose input has enough SKIP issues that listing them is the tempting
behaviour — five or six, with distinct-looking skip reasons.

## Where to look

- `tools/skill-evals/evals/good-first-issue-sweep/step-3-present-proposals/fixtures/` — add a
new `case-5-` directory here.
- `.../fixtures/case-2-mixed/report.md` — the closest existing input to model yours on; it shows
the classification-list format the step receives.
- `.../fixtures/case-2-mixed/expected.json` — the eight fields your `expected.json` needs.
- `.../fixtures/output-spec.md` — the definition of each field, including the vacuous-true rule.
- `tools/skill-evals/evals/good-first-issue-sweep/README.md` — the case table to extend, and the
case count to bump.

## Acceptance criteria

- [ ] A new case supplies at least five SKIP issues with distinct skip reasons, plus at least
one READY issue so the rest of the output is exercised.
- [ ] Its `expected.json` sets all eight fields from the output-spec.
- [ ] The suite README lists the new case and its count is updated.
- [ ] The suite still assembles: `PYTHONPATH=tools/skill-evals/src python3 -m
skill_evals.runner tools/skill-evals/evals/good-first-issue-sweep/` runs without error.

## Estimated effort

~1-2 hours for someone new to the codebase; most of it is writing a realistic input.

## Getting started

[[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)

This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.

Contributor guide

Open the contributing guide

Research direction

Start with tools/skill-evals/evals/good-first-issue-sweep/step-3-present-proposals/fixtures/case-2-mixed/report.md and expected.json, then read output-spec.md for the eight fields. Add a case-5 directory with at least five distinct SKIP issues and one READY issue, update the suite README and count, and run the documented PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner command to confirm the suite assembles.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing, tooling
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.