NVIDIA / NVIDIA/Megatron-LM

[fix] Fix unexpected memory cost of gtp-1shot-RS

Open
#6,361 0 comments 0 reactions 0 assignees View on GitHub
nemotron
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

Image

at 1024-scale (TP2), flip gtp1ShotRS from True to False, total device reduced by ~15GB, and max reserved reduced by 3GB.

Goal: figure out why avoid the unexpected memory cost from `total device`

Related [thread](https://nvidia.slack.com/archives/C8HGQL6D8/p1786384942601809) (internal)

Contributor guide

Open the contributing guide

Research direction

Reproduce the 1024-scale (TP2) comparison with gtp1ShotRS set to True and False, measuring total device memory and maximum reserved memory. Trace the code path controlled by this flag to determine why enabling it adds memory, then verify that the unexpected total-device cost is removed without changing the intended behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.