NVIDIA / NVIDIA/Megatron-LM

[BUG] convert.py fails with ValueError: Default process group has not been initialized

Open
#1,818 1 comment 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

### Description

When attempting to merge Megatron distributed checkpoints using the provided `tools/checkpoint/convert.py` script, the process fails with:

```
ValueError: Default process group has not been initialized, please make sure to call init_process_group.
```

This happens specifically when loading **`torch_dist` distributed checkpoints** (saved with `--use-dist-ckpt`), because the script eventually calls `torch.distributed.get_world_size()` without initializing a default process group.

---

### Command to Reproduce

```bash
CUDA_DEVICE_MAX_CONNECTIONS=1 \
python tools/checkpoint/convert.py \
--model-type BERT \
--loader mcore \
--saver mcore \
--load-dir /path/to/checkpoints \
--save-dir /path/to/merged_checkpoint \
--true-vocab-size 25 \
--position-embedding-type learned_absolute \
--loader-transformer-impl transformer_engine \
--target-tensor-parallel-size 1 \
--target-pipeline-parallel-size 1 \
--saver-transformer-impl transformer_engine
```

---

### Error Traceback

```
Traceback (most recent call last):
File "tools/checkpoint/convert.py", line 246, in
main()
File "tools/checkpoint/convert.py", line 162, in main
loader.load_checkpoint(queue, args)
File "tools/checkpoint/loader_core.py", line 92, in load_checkpoint
raise e
File "tools/checkpoint/loader_core.py", line 89, in load_checkpoint
loader.load()
File "tools/checkpoint/loader_base.py", line 447, in load
self.all_models, self.consumed_train_samples, self.consumed_valid_samples = self.load_model_shards(
File "tools/checkpoint/loader_base.py", line 269, in load_model_shards
all_models.append(get_models_for_pipeline_stage(tp_size, dtype))
File "tools/checkpoint/loader_base.py", line 243, in get_models_for_pipeline_stage
load_checkpoint(model_list, None, None)
File "megatron/training/checkpointing.py", line 1589, in load_checkpoint
state_dict, checkpoint_name, release, ckpt_type = _load_base_checkpoint(
File "megatron/training/checkpointing.py", line 1127, in _load_base_checkpoint
return _load_global_dist_base_checkpoint(
File "megatron/training/checkpointing.py", line 1013, in _load_global_dist_base_checkpoint
state_dict = dist_checkpointing.load(sharded_state_dict, checkpoint_name, load_strategy, strict=args.dist_ckpt_strictness)
File "megatron/core/dist_checkpointing/serialization.py", line 139, in load
local_metadata, global_metadata = determine_global_metadata(sharded_state_dict)
File "megatron/core/dist_checkpointing/validation.py", line 557, in determine_global_metadata
global_metadata = [None] * torch.distributed.get_world_size()
File "torch/distributed/distributed_c10d.py", line 1298, in _get_default_group
raise ValueError("Default process group has not been initialized, please make sure to call init_process_group.")
ValueError: Default process group has not been initialized, please make sure to call init_process_group.
```

---

### Root Cause

* The convert script (`loader_base.py` / `_load_global_dist_base_checkpoint`) calls into `torch.distributed.get_world_size()`.
* However, `torch.distributed.init_process_group()` is never invoked in the conversion path, so no default process group exists.
* `_ConverterFakeProcessGroup` is only a partial workaround for TP/PP groups, but not for the global process group required by `dist_checkpointing`.

---

### Expected Behavior

* `convert.py` should be able to load and merge distributed checkpoints **without requiring the user to manually initialize torch.distributed**.
* At minimum, it should initialize a dummy `init_process_group` (with `rank=0, world_size=1`) when running in single-process conversion.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.