Flaky functional test: hybrid_nemotron_v3_pico_7b_a1b_tp1_ep8_QAD_dgx_h100_1N8G — total loss not bit-exact
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
## Summary
The functional test `hybrid_nemotron_v3_pico_7b_a1b_tp1_ep8_QAD_dgx_h100_1N8G` (`tests/functional_tests/python_test_utils/test_pretraining_regular_pipeline.py::test_regular_pipeline`) is flaky and has repeatedly dequeued PRs from the merge queue.
## Symptom
```
Exact comparison of total loss: FAILED
APPROXIMATE test for metric total loss: PASSED (atol=0, rtol=0.05)
AssertionError: The following metrics failed: total loss
```
The `total loss` is within 5% of golden (approximate check passes) but not bit-exact (deterministic check fails) — run-to-run non-determinism in this hybrid/MoE (ep8) case. Observed failing across multiple independent merge-queue validation runs on unrelated PRs (e.g. #5826), each time dequeuing the PR.
## Temporary mitigation
Scope moved from `[mr, mr-github]` to `[mr-broken, mr-github-broken]` in `tests/test_utils/recipes/h100/mamba.yaml` so it no longer gates merge-queue runs.
## Follow-up
- Investigate the non-determinism source for this config, and either make the run deterministic or refresh/relax the golden (e.g. drop the exact/deterministic check, keep approximate) for `total loss`.
- Re-enable the `mr` / `mr-github` scope once fixed.
Contributor guide
Research direction
Start with tests/functional_tests/python_test_utils/test_pretraining_regular_pipeline.py::test_regular_pipeline and the hybrid configuration referenced by tests/test_utils/recipes/h100/mamba.yaml. Run the named functional test to reproduce the exact-versus-approximate total-loss result, then trace the configuration's deterministic check or golden data. Done means the test is deterministic or its exact check is appropriately adjusted, and the mr/mr-github scope is restored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100