🐛 Weekly regression: Mixtral 8x22B loss divergence and iteration-time slowdown

**Describe the regression**

The H100 weekly functional release test
`mixtral_8x22b_tp2pp8ep8vpp1_release` completed all 150 training iterations but
failed post-training golden-value validation. The steady-state iteration time
is 9.03% above the golden value, and language-model loss is outside the 5%
tolerance at all 31 sampled steps. Both memory metrics passed.

Tag @NVIDIA/mcore-oncall to get oncall's attention to this issue.

**Affected test**

| Field | Value |
|---|---|
| Test case | `tests/functional_tests/test_cases/mixtral/mixtral_8x22b_tp2pp8ep8vpp1_release` |
| Validation test | `tests/functional_tests/python_test_utils/test_pretraining_regular_pipeline.py::test_regular_pipeline` |
| Environment | `dev` on DGX H100 |
| Tested commit | `f258d4fa82de72705f846fb713bbe3b4180f38e8` |
| Training iterations | 150 |

**Metric details**

The validator uses a relative tolerance of 5% for both `iteration-time` and
`lm loss`.

| Metric | Actual | Golden | Difference | Result |
|---|---:|---:|---:|---|
| Median steady-state iteration time | 7.654155 s | 7.020375 s | +9.03% | Failed |
| Language-model loss samples | 31 of 31 outside tolerance | — | Minimum +21.84%; maximum +332.91% | Failed |
| `mem-allocated-bytes` | Within tolerance | Within tolerance | — | Passed |
| `mem-max-allocated-bytes` | Within tolerance | Within tolerance | — | Passed |

Representative language-model loss samples:

| Step | Actual | Golden | Difference |
|---:|---:|---:|---:|
| 1 | 2.18759 | 1.73932 | +25.77% |
| 60 | 2.36206 | 1.58250 | +49.26% |
| 145 | 6.52672 | 1.50763 | +332.91% |
| 150 | 6.01281 | 1.53412 | +291.94% |

The loss starts above the golden curve and diverges substantially during the
run instead of converging near 1.5.

**Error**

```text
Actual values: 7.654154999999999
Golden values: 7.020375
Approximate comparison of iteration-time: FAILED
Approximate comparison of lm loss: FAILED
APPROXIMATE test for metric mem-allocated-bytes: PASSED
APPROXIMATE test for metric mem-max-allocated-bytes: PASSED

FAILED tests/functional_tests/python_test_utils/test_pretraining_regular_pipeline.py::test_regular_pipeline
AssertionError: The following metrics failed: iteration-time, lm loss
```

**Steps/Code to reproduce bug**

Run the `mixtral_8x22b_tp2pp8ep8vpp1_release` functional release recipe in the
`dev` DGX H100 environment. After training, validate the generated TensorBoard
metrics with:

```bash
uv run pytest -s \
tests/functional_tests/python_test_utils/test_pretraining_regular_pipeline.py \
--golden-values-path tests/functional_tests/test_cases/mixtral/mixtral_8x22b_tp2pp8ep8vpp1_release/golden_values_dev_dgx_h100.json \
--actual-values-path \
--train-iters 150 \
--model-config-path tests/functional_tests/test_cases/mixtral/mixtral_8x22b_tp2pp8ep8vpp1_release/model_config.yaml \
--allow-nondeterministic-algo
```

**Additional context**

- Training itself completed; rank 0 failed only during the post-training metric
comparison.
- The Mixtral 8x7B iteration-time-only weekly failure is tracked separately in
#5456.