[BUG] convert.py fails with ValueError: Default process group has not been initialized
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
### Description
When attempting to merge Megatron distributed checkpoints using the provided `tools/checkpoint/convert.py` script, the process fails with:
```
ValueError: Default process group has not been initialized, please make sure to call init_process_group.
```
This happens specifically when loading **`torch_dist` distributed checkpoints** (saved with `--use-dist-ckpt`), because the script eventually calls `torch.distributed.get_world_size()` without initializing a default process group.
---
### Command to Reproduce
```bash
CUDA_DEVICE_MAX_CONNECTIONS=1 \
python tools/checkpoint/convert.py \
--model-type BERT \
--loader mcore \
--saver mcore \
--load-dir /path/to/checkpoints \
--save-dir /path/to/merged_checkpoint \
--true-vocab-size 25 \
--position-embedding-type learned_absolute \
--loader-transformer-impl transformer_engine \
--target-tensor-parallel-size 1 \
--target-pipeline-parallel-size 1 \
--saver-transformer-impl transformer_engine
```
---
### Error Traceback
```
Traceback (most recent call last):
File "tools/checkpoint/convert.py", line 246, in
main()
File "tools/checkpoint/convert.py", line 162, in main
loader.load_checkpoint(queue, args)
File "tools/checkpoint/loader_core.py", line 92, in load_checkpoint
raise e
File "tools/checkpoint/loader_core.py", line 89, in load_checkpoint
loader.load()
File "tools/checkpoint/loader_base.py", line 447, in load
self.all_models, self.consumed_train_samples, self.consumed_valid_samples = self.load_model_shards(
File "tools/checkpoint/loader_base.py", line 269, in load_model_shards
all_models.append(get_models_for_pipeline_stage(tp_size, dtype))
File "tools/checkpoint/loader_base.py", line 243, in get_models_for_pipeline_stage
load_checkpoint(model_list, None, None)
File "megatron/training/checkpointing.py", line 1589, in load_checkpoint
state_dict, checkpoint_name, release, ckpt_type = _load_base_checkpoint(
File "megatron/training/checkpointing.py", line 1127, in _load_base_checkpoint
return _load_global_dist_base_checkpoint(
File "megatron/training/checkpointing.py", line 1013, in _load_global_dist_base_checkpoint
state_dict = dist_checkpointing.load(sharded_state_dict, checkpoint_name, load_strategy, strict=args.dist_ckpt_strictness)
File "megatron/core/dist_checkpointing/serialization.py", line 139, in load
local_metadata, global_metadata = determine_global_metadata(sharded_state_dict)
File "megatron/core/dist_checkpointing/validation.py", line 557, in determine_global_metadata
global_metadata = [None] * torch.distributed.get_world_size()
File "torch/distributed/distributed_c10d.py", line 1298, in _get_default_group
raise ValueError("Default process group has not been initialized, please make sure to call init_process_group.")
ValueError: Default process group has not been initialized, please make sure to call init_process_group.
```
---
### Root Cause
* The convert script (`loader_base.py` / `_load_global_dist_base_checkpoint`) calls into `torch.distributed.get_world_size()`.
* However, `torch.distributed.init_process_group()` is never invoked in the conversion path, so no default process group exists.
* `_ConverterFakeProcessGroup` is only a partial workaround for TP/PP groups, but not for the global process group required by `dist_checkpointing`.
---
### Expected Behavior
* `convert.py` should be able to load and merge distributed checkpoints **without requiring the user to manually initialize torch.distributed**.
* At minimum, it should initialize a dummy `init_process_group` (with `rank=0, world_size=1`) when running in single-process conversion.
Contributor guide
Assessment
This issue has not been assessed yet.