[Bug] TestGemma4MoE::test_bf16 fails deterministically in pre-merge DGX_H100-4_GPUs-AutoDeploy-1: Not registered specs (bf16/auto)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Summary
accuracy/test_llm_api_autodeploy.py::TestGemma4MoE::test_bf16 fails deterministically with:
ValueError: Not registered specs: {'dtype': 'auto', 'quant_algo': None, 'kv_cache_quant_algo': None, 'spec_dec_algo': None, ...}
The accuracy reference files (mmlu/gsm8k/mmmu yaml) contain google/gemma-4-26B-A4B-it entries, but none matching the plain bf16/auto config this test requests.
Impact
- Stage
DGX_H100-4_GPUs-AutoDeploy-1is pre-merge, so any unrelated PR whose run selects this stage fails CI (observed on PR #15730: L0_MergeRequest_PR builds 47313 and 47345, base = current main in both cases). - Because the stage is pre-merge-only, the post-merge waive automation never sees it — no auto-waive will come.
Ask
Either register the missing spec for the bf16/auto config, or waive the test until it is fixed. Also reported on #12710 (test origin).
cc @NVIDIA/trt-llm-autodeploy-devs
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with accuracy/test_llm_api_autodeploy.py::TestGemma4MoE::test_bf16 and inspect the mmlu, gsm8k, and mmmu YAML reference entries for google/gemma-4-26B-A4B-it. Compare those registrations with the requested bf16/auto configuration and determine whether the missing spec should be registered or the test waived; done means the pre-merge stage no longer fails deterministically.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100