Held-out shape injection misses relocated harnesses and renamed vLLM runners

Summary

Several harness relocations and two vLLM runner renames break the repository’s held-out shape injection workflow. src.held_out still assumes the old injection targets, so held-out evaluation can miss the files that correctness and performance actually run.

Introduced on main by b2bd34da (fix(tasks): consolidate benchmark correctness and harness hardening, PR #105). CPU unit tests still pass because they only exercise the old TEST_SHAPES / root pytorch_code_* contract.

What is broken

Held-out injection (src/held_out/injection.py, prompts in generate_heldout.py, and src/held_out/README.md) still targets:

• vLLM: TEST_SHAPES = [...] in scripts/task_runner.py
• hip2hip/torch2hip gpumode: get_inputs() in workspace-root pytorch_code_module/*.py and pytorch_code_functional/*_func.py

Harness relocations (hip2hip gpumode)

Scored harnesses now read relocated references, while root copies still exist for some tasks. Injecting into the documented root paths therefore either fails or silently updates stale files:

vLLM runner marker renames

These two runners no longer define TEST_SHAPES, which is the only vLLM marker replace_test_shapes() looks for:

• tasks/triton2triton/vllm/triton_moe_mmk/scripts/task_runner.py: TEST_SHAPES → PERFORMANCE_SHAPES / CORRECTNESS_SHAPES
• tasks/triton2triton/vllm/triton_scaled_mm/scripts/task_runner.py: TEST_SHAPES → PERFORMANCE_SHAPES / CORRECTNESS_CASES

Injection into those tasks fails (or existing held_out_shapes.yaml becomes a no-op).

Why tests did not catch this

• tests/test_held_out.py uses synthetic sources, not live task layouts.
• tests/test_quality_loop_runner_contracts.py copies scripts/task_runner.py and injects TEST_SHAPES for tasks that still use that name.

Suggested fix

• Resolve injection files from each task’s config.yaml (--py_modu_file / --py_func_file / runner path), not from hardcoded layout assumptions.
• Support the current vLLM shape markers (TEST_SHAPES, CORRECTNESS_SHAPES, CORRECTNESS_CASES, and related performance lists) without scoring a different set than correctness/performance.
• Add regression tests that apply injection against the real relocated hip2hip tasks and the two renamed vLLM runners, then assert the scored files change.

Start with src/held_out/injection.py, generate_heldout.py, and the task config.yaml files that define injection paths and runner markers. Compare the real relocated hip2hip harnesses and the two vLLM task_runner.py files, then run tests/test_held_out.py and tests/test_quality_loop_runner_contracts.py while adding regression coverage. Done means injection changes the scored files and supports the listed current shape markers without changing the correctness or performance sets.