NVIDIA-NeMo / NVIDIA-NeMo/RL

NeMo-RL vs megatron-core `main`: current functional-test breakage

Open
#3,365 0 comments 0 reactions 1 assignee Claimed by @terrykong View on GitHub
accuracy Automation bug
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

## L1 functional tests against megatron-core `main`

Last pass: 2026-08-06 00:30 UTC · 6/7 green.

This issue is maintained automatically and edited in place. It tracks breakage that is **not** caused by any particular Megatron-LM pull request, so that labeled PRs do not each have to rediscover it.

### Stack under test

- megatron-core: `main` @ `19c50e86`
- Megatron-Bridge: `2940a635`
- NeMo-RL: `main` plus the pending fixes below @ `86774472`

### Current status

| Test | Status | Assessment |
| --- | --- | --- |
| `grpo_megatron_generation` | ✅ pass | — |
| `grpo_megatron_generation_non_colocated` | ✅ pass | — |
| `grpo_megatron_generation_async` | ✅ pass | — |
| `grpo_megatron_generation_colocated_async` | ✅ pass | — |
| `grpo_megatron_generation_async_gym` | ✅ pass | — |
| `grpo_megatron_generation_topp_topk` | ❌ fail | max(data["train/token_mult_prob_error"]) < 1.06 (measured 1.078047513961792). Known issue `topp-topk-token-mult-prob-error-over-threshold` (see below). |
| `grpo_megatron_generation_colocated_gym` | ✅ pass | — |

### Diagnoses

**`topp-topk-token-mult-prob-error-over-threshold`** — 1 test(s): `grpo_megatron_generation_topp_topk`

Pre-existing on megatron-core main, not caused by any labeled PR. grpo_megatron_generation_topp_topk runs to completion but max(train/token_mult_prob_error) lands at 1.078047513961792 against a < 1.06 threshold. The value is bit-identical on Megatron-LM#5382 head eb01b689 and on main 3aee84c3 (both with the Bridge FSDP fix), so it is a deterministic train/generation logprob mismatch in the top-p/top-k sampling path rather than run-to-run noise. Marginal at ~1.7% over. No fix raised; needs an owner to decide whether the sampling path regressed or the threshold is stale. **No fix raised yet.**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.