NVIDIA / NVIDIA/Megatron-LM

Flaky functional test: hybrid_nemotron_v3_pico_7b_a1b_tp1_ep8_QAD_dgx_h100_1N8G — total loss not bit-exact

Open
#5,832 0 comments 0 reactions 0 assignees View on GitHub
bug flaky-test
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

## Summary

The functional test `hybrid_nemotron_v3_pico_7b_a1b_tp1_ep8_QAD_dgx_h100_1N8G` (`tests/functional_tests/python_test_utils/test_pretraining_regular_pipeline.py::test_regular_pipeline`) is flaky and has repeatedly dequeued PRs from the merge queue.

## Symptom

```
Exact comparison of total loss: FAILED
APPROXIMATE test for metric total loss: PASSED (atol=0, rtol=0.05)
AssertionError: The following metrics failed: total loss
```

The `total loss` is within 5% of golden (approximate check passes) but not bit-exact (deterministic check fails) — run-to-run non-determinism in this hybrid/MoE (ep8) case. Observed failing across multiple independent merge-queue validation runs on unrelated PRs (e.g. #5826), each time dequeuing the PR.

## Temporary mitigation

Scope moved from `[mr, mr-github]` to `[mr-broken, mr-github-broken]` in `tests/test_utils/recipes/h100/mamba.yaml` so it no longer gates merge-queue runs.

## Follow-up

- Investigate the non-determinism source for this config, and either make the run deterministic or refresh/relax the golden (e.g. drop the exact/deterministic check, keep approximate) for `total loss`.
- Re-enable the `mr` / `mr-github` scope once fixed.

Contributor guide

Open the contributing guide

Research direction

Start with tests/functional_tests/python_test_utils/test_pretraining_regular_pipeline.py::test_regular_pipeline and the hybrid configuration referenced by tests/test_utils/recipes/h100/mamba.yaml. Run the named functional test to reproduce the exact-versus-approximate total-loss result, then trace the configuration's deterministic check or golden data. Done means the test is deterministic or its exact check is appropriately adjusted, and the mr/mr-github scope is restored.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.