NVIDIA / NVIDIA/TensorRT-LLM

[Bug] test_dflash_qwen3_5_4b flaky-marginal in DGX_H100-PyTorch-5: mean accepted 0.9909 < 1.0, alternating [True]/[False] variants

Open
#16,277 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Speculative Decoding Testing
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

Summary

unittest/_torch/speculative/hw_agnostic/test_dflash.py::test_dflash_qwen3_5_4b fails whenever the pre-merge stage DGX_H100-PyTorch-5 runs, with:

AssertionError: mean avg accepted 0.99 < threshold 1.0 (per request: ['1.37', '0.57', '1.03'])
assert 0.9908895492553711 >= 1.0

Identical numeric outcome (0.9908895) across independent runs, but the failing parametrization alternates: on L0_MergeRequest_PR build 47135 [False] failed while [True] passed; on build 47357 [True] failed while [False] passed (both on PR #15730, an unrelated VisualGen test PR, base = current main). Looks like whichever variant runs in a particular session position/state lands at 0.9909 against the exact-1.0 threshold.

Impact

Pre-merge blocker for unrelated PRs whose runs select the stage; the threshold of exactly 1.0 leaves zero margin.

Ask

Relax the threshold / deflake, or waive both variants until fixed. Related platform-level waives already exist for TestQwen3_5_4B::test_dflash (nvbug 6273850).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with unittest/_torch/speculative/hw_agnostic/test_dflash.py::test_dflash_qwen3_5_4b and reproduce both [True] and [False] parametrizations in the DGX_H100-PyTorch-5 context. Check why the mean accepted value is consistently 0.9908895 against the 1.0 threshold; done means the pre-merge stage no longer flakes or both variants are appropriately waived.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.