rossoctl / rossoctl/serverless-harness

Plan B Task 5 follow-ups: measurement driver polish + matplotlib collection caveat

Open
#130 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
1
Forks
7
Avg merge
12h 59m
Merged PRs (30d)
71

Description

Follow-up items from Plan B Task 5 (#129, SWE-bench runtime measurement). All non-blocking; documented for later.

1. MEASURE_SAMPLES>1 status-label aggregation (deploy/knative/measure-swebench-runtimes.sh)

With more than one sample per instance, the per-sample status label reflects only the last sample, and median could mix a real runtime with the timeout-cap value. Purely cosmetic at the default MEASURE_SAMPLES=1 (the only exercised path). If multi-sample is ever used, treat any timeout sample as dominating and aggregate ms only within a single status class.

2. CI shellcheck for the measurement driver

Add deploy/knative/measure-swebench-runtimes.sh to the CI shellcheck coverage (it was shellcheck-clean at authoring, but isn't gated in CI).

3. matplotlib collection-as-error under newer setuptools

matplotlib__matplotlib-24177's measured runtime is import+collection only: its pytest collection errors on a DeprecationWarning-as-error at base_commit (newer setuptools in the baked env), so 0 tests are collected. The recorded time is faithful to the gold command but undervalues the instance. If a truer matplotlib number is wanted, relax the warning filter for the measurement run (e.g. PYTHONWARNINGS=default) — note this deviates from the exact gold test_cmd.

4. test_cmd-as-list defensive branch (scripts/gen_swebench_deck.py)

The list→string join branch is unexercised with swebench 4.1.0 (returns str); kept for forward-compat. Add a unit case if a future swebench version returns a list.

Assisted-By: Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading deploy/knative/measure-swebench-runtimes.sh and scripts/gen_swebench_deck.py, then inspect the CI shellcheck configuration and existing tests. Separate the four follow-ups: multi-sample status aggregation, shellcheck coverage, the matplotlib warning-filter decision, and coverage for list-valued test_cmd. Done means each selected behavior is implemented or tested without changing the default measurement path unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
matplotlib, python, shell
Domain
ci-cd, performance, testing, tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.