NVIDIA-NeMo / NVIDIA-NeMo/RL

Make MBridge submodule bump conflict-free (flashinfer-jit-cache, nvidia-cutlass-dsl pins)

Open
#4,141 0 comments 0 reactions 1 assignee Claimed by @kajalj22 View on GitHub
Automation bug
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

## Summary

RL's `mcore` extra hand-pins a couple of packages that Megatron-Bridge either already owns or dictates the version of. Whenever Megatron-Bridge bumps those versions, RL's duplicate pins go stale and `uv lock` breaks — as happened this week (root-caused and unblocked in #4139 and #4140). This issue tracks removing the duplication itself so a submodule bump can't conflict on these anymore, rather than re-fixing it each time it happens.

## Root cause

`megatron-bridge` is a `uv` workspace member of RL, so its own `pyproject.toml` deps flow directly into RL's lockfile. RL's `mcore` extra separately hand-pins a few packages Megatron-Bridge doesn't provide — those duplicated/adjacent pins are what breaks every time Megatron-Bridge moves a version:

```mermaid
flowchart TD
A["RL mcore extra"] --> B["megatron-bridge[te,ssm]
(workspace member)"]
A --> C["Pins RL keeps direct"]

B --> D["megatron-bridge's own deps"]
D --> E["megatron-core
fully inherited"]
D --> F["flashinfer-python / -cubin
fully inherited since #4140"]

C --> G["flashinfer-jit-cache
not installed by megatron-bridge at all"]
C --> H["nvidia-cutlass-dsl==4.6.2
megatron-bridge explicitly excludes it
(sys_platform == 'never')"]
C --> I["transformers==5.12.1
redundant exact pin; megatron-bridge
already installs it (>=5.12.1,<=5.15.0)"]
C --> J["tilelang==0.1.9
redundant exact pin; megatron-bridge's
ssm extra already installs it (>=0.1.9)"]

F -.->|"drift risk"| G
F -.->|"drift risk"| H
```

This is the same pattern already solved for `megatron-core` in #2136 / #3235 — just not yet extended to these four.

## The four at-risk pins, and what to do about each

| Pin | RL declares | Megatron-Bridge declares | Resolution |
|---|---|---|---|
| `transformers` | `==5.12.1` (exact) | `>=5.12.1,<=5.15.0` (range) | **Drop the pin from RL entirely.** Megatron-Bridge already installs it; RL's copy is pure duplication, same as `megatron-core` — no reason to pin a narrower value here at all. |
| `tilelang` | `==0.1.9` (exact) | `ssm` extra: `>=0.1.9` (floor) | **Drop the pin from RL entirely**, same reasoning. |
| `flashinfer-jit-cache` | `==0.6.18.post1` (exact) | not declared at all | **Megatron-Bridge should provide this.** It must track `flashinfer-python`'s exact version for correctness (mismatched kernel providers = wrong kernels) — there's no `uv`/PEP 621 way to express "version of A must equal resolved version of B" from RL's side. Once Megatron-Bridge adds it alongside `flashinfer-python`/`-cubin` (which it already keeps in lockstep), RL can drop its pin the same way it already dropped `flashinfer-python`/`-cubin`. |
| `nvidia-cutlass-dsl` | `==4.6.2` (exact) | explicitly excluded (`sys_platform == 'never'`, comment: "rely on system site packages install") | **Needs more context from the Megatron-Bridge team before deciding.** The exclusion looks deliberate, so this isn't a clear-cut "just install it" ask — need to understand why it's excluded there before proposing RL drop or relax its own pin. |

## Action items
- [ ] RL: drop the `transformers==5.12.1` pin from `mcore` extra entirely — already flows in from megatron-bridge
- [ ] RL: drop the `tilelang==0.1.9` pin from `mcore` extra entirely — already flows in from megatron-bridge
- [ ] Ask Megatron-Bridge to add `flashinfer-jit-cache` to its unconditional deps, in lockstep with `flashinfer-python`/`-cubin`, so RL can drop its own pin
- [ ] Ask Megatron-Bridge team for more context on why `nvidia-cutlass-dsl` is excluded (`sys_platform == 'never'`) before deciding how to handle RL's pin

## Evidence
`nemo-rl-nightly-mbridge-bump` failed 5 consecutive nights (2026-09-11 → 09-15) from exactly this duplication drifting; see #4139/#4140 for the full trace.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.