NVIDIA / NVIDIA/Megatron-LM

GDN packed sequence support landing in Nemo containers

Open
#5,044 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

Currently tracking packed sequence support for GDN (Gated Delta Net), specifically PRs #2644 and #2645 (e.g., https://github.com/NVIDIA/Megatron-LM/pull/2644) in megatron-lm being merged to mainline and the Q2 roadmap lists it as targeting the 26.06 release. Some follow up questions:

1. The fix landed on main on May 14, after the core_v0.17.0 tag (April 14). Will it be cherry-picked into a 0.17.x patch release, or will it ship as part of 0.18?
2. Is 26.06 / the June NeMo container expected to include 0.17 or 0.18?
3.  from the PR: it requires cudnn>=9.19.0 for Qwen3.5 THD training on Hopper to avoid grad NaN. Can we confirm which NeMo container will have both cudann >= 9.19.0 and the fix?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.