NVIDIA-NeMo / NVIDIA-NeMo/Automodel

Custom module weights not synchronized across TP ranks

Open
#1,291 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

  • #1292 by @akoumpa — closed without merging
bug
Dominant language
Python
Stars
963
Forks
318
Avg merge
3d 20h
Merged PRs (30d)
143

Description

Describe the bug

When using Tensor Parallelism (tp_size > 1), custom EmbeddingBlock weights are not synchronized across TP ranks. Each rank initializes with a different RNG state, producing divergent weights on a replicated (non-sharded) layer. This results in incorrect loss that scales inversely with tp_size.

Steps/Code to reproduce bug

Environment: NeMo AutoModel on Databricks, T4 / L40S GPUs

  1. Define a custom HF-compatible model with a custom EmbeddingBlock
  2. Train with NeMo AutoModel, varying tp_size

Expected behavior

All TP ranks should have identical weights for replicated layers after initialization. Initial loss should be consistent regardless of tp_size.

Additional context

  1. Observed Behavior:
tp_size Actual Loss Expected Loss Result
1 ~11.75 ~11.75 Correct
2 ~5.83 ~11.75 Wrong
8 ~1.45 ~11.75 Wrong
  1. Debug output with tp_size=8
    Embedding weights not synchronized across TP ranks despite correct non-sharded shape (112146, 128):
# Debug output with tp_size=8 show each TP rank has completely different embedding weights:
[GPU 0] Sample values [0,0,:4]: [0.0183, -0.0159, -0.0233, -0.0464]
[GPU 1] Sample values [0,0,:4]: [-0.0199, -0.0315, 0.00397, 0.0157]
[GPU 2] Sample values [0,0,:4]: [0.00778, 0.00723, 0.0569, -0.00516]
[GPU 3] Sample values [0,0,:4]: [0.0231, -0.0182, 0.0223, -0.000319]
[GPU 4] Sample values [0,0,:4]: [-0.00958, -0.0165, 0.00224, -0.0167]
[GPU 5] Sample values [0,0,:4]: [-0.00806, -0.0544, -0.00519, -0.0107]
[GPU 6] Sample values [0,0,:4]: [-0.0253, 0.0156, -0.0393, -0.00323]
[GPU 7] Sample values [0,0,:4]: [0.0226, 0.0182, -0.00143, -0.00595]

NeMo AutoModel only automatically synchronizes weights for recognized HuggingFace model components. Custom modules like EmbeddingBlock are initialized independently on each TP rank:

# GPU 0: EmbeddingBlock() → random init with seed A → weights A
# GPU 1: EmbeddingBlock() → random init with seed B → weights B  (DIFFERENT!)
# GPU 2: EmbeddingBlock() → random init with seed C → weights C  (DIFFERENT!)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source files or tests are named. Reproduce the custom EmbeddingBlock with varying tp_size, then inspect NeMo AutoModel's initialization and weight-synchronization path for replicated modules; done means identical weights across TP ranks and consistent initial loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.