NVIDIA / NVIDIA/Megatron-LM

[GTP] Documentation update

Open
#6,236 0 comments 0 reactions 1 assignee Claimed by @fanshiqing View on GitHub
nemotron
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 3h
Merged PRs (30d)
272

Description

What's need to update later:
- Best practices related
- `export NVTE_FWD_LAYERNORM_SM_MARGIN=32`, `export NVTE_BWD_LAYERNORM_SM_MARGIN=32` to make LN fwd/bwd can be overlapped with async GTP AG/RS
- add `MEGATRON_LOGGING_LEVEL=20` if you want to check the GTP prefetch chain construction details
- Don't set `CUDA_DEVICE_MAX_CONNECTIONS` for Blackwell+
- `--gtp-remat-opt-in-modules moe_latent_proj` when the latent size is large enough to amortize the all-gather for more memory saving.
- `export NVTE_INF_LAYERNORM_SM_MARGIN=32` for `gdp_in_proj`
- required for PartialCG:
- `export PYTORCH_CUDA_ALLOC_CONF=graph_capture_record_stream_reuse:True` for memory saving
- export CUDA_GRAPHS_USE_NODE_PRIORITY=1
- export CUDA_DEVICE_MAX_CONNECTIONS=32
- export NVTE_INF_LAYERNORM_SM_MARGIN=32
- GTP+DCP related dtype convetion explaination now that we have changed the workflow for bit-wise matching.
- tune gtp realted nccl comms: `--nccl-communicator-config-path `

- Latest perf / scalability on nemotron-next

- Update supporting matrix for now

- Others...

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.