rossoctl / rossoctl/serverless-harness
Plan B Task 5 follow-ups: measurement driver polish + matplotlib collection caveat
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 7
- Avg merge
- 12h 59m
- Merged PRs (30d)
- 71
Description
Follow-up items from Plan B Task 5 (#129, SWE-bench runtime measurement). All non-blocking; documented for later.
1. MEASURE_SAMPLES>1 status-label aggregation (deploy/knative/measure-swebench-runtimes.sh)
With more than one sample per instance, the per-sample status label reflects only the last sample, and median could mix a real runtime with the timeout-cap value. Purely cosmetic at the default MEASURE_SAMPLES=1 (the only exercised path). If multi-sample is ever used, treat any timeout sample as dominating and aggregate ms only within a single status class.
2. CI shellcheck for the measurement driver
Add deploy/knative/measure-swebench-runtimes.sh to the CI shellcheck coverage (it was shellcheck-clean at authoring, but isn't gated in CI).
3. matplotlib collection-as-error under newer setuptools
matplotlib__matplotlib-24177's measured runtime is import+collection only: its pytest collection errors on a DeprecationWarning-as-error at base_commit (newer setuptools in the baked env), so 0 tests are collected. The recorded time is faithful to the gold command but undervalues the instance. If a truer matplotlib number is wanted, relax the warning filter for the measurement run (e.g. PYTHONWARNINGS=default) — note this deviates from the exact gold test_cmd.
4. test_cmd-as-list defensive branch (scripts/gen_swebench_deck.py)
The list→string join branch is unexercised with swebench 4.1.0 (returns str); kept for forward-compat. Add a unit case if a future swebench version returns a list.
Assisted-By: Claude Code
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading deploy/knative/measure-swebench-runtimes.sh and scripts/gen_swebench_deck.py, then inspect the CI shellcheck configuration and existing tests. Separate the four follow-ups: multi-sample status aggregation, shellcheck coverage, the matplotlib warning-filter decision, and coverage for list-valued test_cmd. Done means each selected behavior is implemented or tested without changing the default measurement path unexpectedly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- matplotlib, python, shell
- Domain
- ci-cd, performance, testing, tooling
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100