test(BACKEND-GATE-ROCM-SGLANG): calibrate a Qwen3-4B correctness proposal
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-GATE-ROCM-SGLANG
Owner: current Strix campaign operator. Parent: #3053. Spec: .agents/specs/strix-qwen3-4b-distributional-calibration.md (commit before implementation).
Pinned production vLLMe126687a9a828d513c01a07cd69f025f27d63280 showed repeated greedy token differences on Qwen3-4B BF16 at c1 and c4. Qualification802b32c9, result SHA1d518a475a79947d9921be92e9893ed74a4831359063284524b3c32a0db4f400, remains NOT_ACCEPTED. User approved scoping a distributional correctness proposal, not numerical tolerances or replacement of the strict gate.
Implement reviewed diagnostic score capture through public incremental generation, not whole-prefix prefill scoring labeled as decode. Ours6e3cbfb exposes a public logits callback; pinned vLLM exposes named importable LogitsProcessor classes, JSON-compatible extra_args and full-vocabulary sample logprobs. Verify these routes execute with the campaign production runner/graphs before calibration. No serialized callables or eager correctness/performance substitution. Compare original prompts plus identical consumed continuation prefixes; capture scores before any forcing. Record actual scheduler shapes and instrumentation equivalence, not requested concurrency alone.
Predeclare oracle-only calibration and held-out validation before viewing candidate scores. Separate c1/c4 and first-prefill/incremental phases. Proposed thresholds, sample counts and statistical rule require subsequent explicit ratification. Existing CUDA/27B/quantized tolerances are not transferable. Stable oracle behavior remains token-exact. Enforce hard failures for prefix/tokenizer/dtype/model changes, nonfinite/missing scores, wrong execution phase, swapped rows, post-forcing capture and skipped callbacks; include corruption controls. A forced output match is not a correctness pass.
Acceptance: source-bound diagnostic design, test-first implementation, independent mutation review, operator leased equivalence and oracle-only evidence, and a written proposed gate with held-out results. This issue does not itself authorize adoption of a tolerance or acceptance of speed. Stop if instrumentation changes the relevant numerical path or production capture cannot be established; name the exact unresolved interface.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with .agents/specs/strix-qwen3-4b-distributional-calibration.md, then inspect the public logits callback in Ours6e3cbfb and the pinned vLLM named LogitsProcessor interfaces. Verify those routes with the campaign production runner and graphs before implementing test-first diagnostics. Done means reviewed source-bound capture, mutation controls, oracle-only calibration evidence, and a written held-out proposal without changing the strict gate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100