NVIDIA-NeMo / NVIDIA-NeMo/Automodel
Custom module weights not synchronized across TP ranks
Nobody has claimed this yet.
- #1292 by @akoumpa — closed without merging
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
Describe the bug
When using Tensor Parallelism (tp_size > 1), custom EmbeddingBlock weights are not synchronized across TP ranks. Each rank initializes with a different RNG state, producing divergent weights on a replicated (non-sharded) layer. This results in incorrect loss that scales inversely with tp_size.
Steps/Code to reproduce bug
Environment: NeMo AutoModel on Databricks, T4 / L40S GPUs
- Define a custom HF-compatible model with a custom
EmbeddingBlock - Train with NeMo AutoModel, varying
tp_size
Expected behavior
All TP ranks should have identical weights for replicated layers after initialization. Initial loss should be consistent regardless of tp_size.
Additional context
- Observed Behavior:
tp_size |
Actual Loss | Expected Loss | Result |
|---|---|---|---|
| 1 | ~11.75 | ~11.75 | Correct |
| 2 | ~5.83 | ~11.75 | Wrong |
| 8 | ~1.45 | ~11.75 | Wrong |
- Debug output with
tp_size=8
Embedding weights not synchronized across TP ranks despite correct non-sharded shape(112146, 128):
# Debug output with tp_size=8 show each TP rank has completely different embedding weights:
[GPU 0] Sample values [0,0,:4]: [0.0183, -0.0159, -0.0233, -0.0464]
[GPU 1] Sample values [0,0,:4]: [-0.0199, -0.0315, 0.00397, 0.0157]
[GPU 2] Sample values [0,0,:4]: [0.00778, 0.00723, 0.0569, -0.00516]
[GPU 3] Sample values [0,0,:4]: [0.0231, -0.0182, 0.0223, -0.000319]
[GPU 4] Sample values [0,0,:4]: [-0.00958, -0.0165, 0.00224, -0.0167]
[GPU 5] Sample values [0,0,:4]: [-0.00806, -0.0544, -0.00519, -0.0107]
[GPU 6] Sample values [0,0,:4]: [-0.0253, 0.0156, -0.0393, -0.00323]
[GPU 7] Sample values [0,0,:4]: [0.0226, 0.0182, -0.00143, -0.00595]
NeMo AutoModel only automatically synchronizes weights for recognized HuggingFace model components. Custom modules like EmbeddingBlock are initialized independently on each TP rank:
# GPU 0: EmbeddingBlock() → random init with seed A → weights A
# GPU 1: EmbeddingBlock() → random init with seed B → weights B (DIFFERENT!)
# GPU 2: EmbeddingBlock() → random init with seed C → weights C (DIFFERENT!)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files or tests are named. Reproduce the custom EmbeddingBlock with varying tp_size, then inspect NeMo AutoModel's initialization and weight-synchronization path for replicated modules; done means identical weights across TP ranks and consistent initial loss.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100