Add a multi-SKIP case so the gfi-sweep summary-only rule can actually fail
- Dominant language
- Python
- Stars
- 92
- Forks
- 92
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 104
Description
## Summary
In the `good-first-issue-sweep` Step 3 eval suite, the field that checks SKIP issues are shown
as a count summary is `true` in all four cases, three of them vacuously. Add a case with
several SKIP issues so the rule has something to fail on.
## Background
`skip_count_shown_as_summary_only` encodes a real presentation rule: SKIP issues must appear as
a count line, not as a list of titles and reasons. But of the four existing cases, only
`case-2-mixed` has any SKIP issues at all — and it has one. The output-spec says that when
there are no SKIP issues the field is `true` "vacuously satisfied", so the other three expect
`true` for free.
That means a model that always answers `true` passes the whole suite on this field, including
one that helpfully lists every skipped issue with its reason. The project's own guidance in
`docs/education/eval-driven-development.md` names this pattern under "All your cases expect the
same value".
The fix is a new case whose input has enough SKIP issues that listing them is the tempting
behaviour — five or six, with distinct-looking skip reasons.
## Where to look
- `tools/skill-evals/evals/good-first-issue-sweep/step-3-present-proposals/fixtures/` — add a
new `case-5-` directory here.
- `.../fixtures/case-2-mixed/report.md` — the closest existing input to model yours on; it shows
the classification-list format the step receives.
- `.../fixtures/case-2-mixed/expected.json` — the eight fields your `expected.json` needs.
- `.../fixtures/output-spec.md` — the definition of each field, including the vacuous-true rule.
- `tools/skill-evals/evals/good-first-issue-sweep/README.md` — the case table to extend, and the
case count to bump.
## Acceptance criteria
- [ ] A new case supplies at least five SKIP issues with distinct skip reasons, plus at least
one READY issue so the rest of the output is exercised.
- [ ] Its `expected.json` sets all eight fields from the output-spec.
- [ ] The suite README lists the new case and its count is updated.
- [ ] The suite still assembles: `PYTHONPATH=tools/skill-evals/src python3 -m
skill_evals.runner tools/skill-evals/evals/good-first-issue-sweep/` runs without error.
## Estimated effort
~1-2 hours for someone new to the codebase; most of it is writing a realistic input.
## Getting started
[[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)
This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.
Contributor guide
Research direction
Start with tools/skill-evals/evals/good-first-issue-sweep/step-3-present-proposals/fixtures/case-2-mixed/report.md and expected.json, then read output-spec.md for the eight fields. Add a case-5 directory with at least five distinct SKIP issues and one READY issue, update the suite README and count, and run the documented PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner command to confirm the suite assembles.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing, tooling
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100