megatron.core.dist_checkpointing.core.CheckpointingException: Invalid access pattern: 36 ShardedObject are missing
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
On latest main with cleared cache and settings:
when using megatron and
tensor_model_parallel_size: 4
expert_tensor_parallel_size: 1
expert_model_parallel_size: 4
pipeline_model_parallel_size: 4
the checkpoint conversion to from megatron to hf once again fails with:
[rank0]: sharded_state_dict, missing_keys, unexpected_keys = validate_integrity_and_strict_load(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/vast/home/stan/git/RL/3rdparty/Megatron-LM-workspace/Megatron-LM/megatron/core/dist_checkpointing/valing/validation.py", line 201, in validate_integrity_and_strict_load
[rank0]: validate_sharding_integrity(global_metadata)
[rank0]: File "/mnt/vast/home/stan/git/RL/3rdparty/Megatron-LM-workspace/Megatron-LM/megatron/core/dist_checkpointing/valing/validation.py", line 442, in validate_sharding_integrity
[rank0]: _validate_objects_for_key(shardings)
[rank0]: File "/mnt/vast/home/stan/git/RL/3rdparty/Megatron-LM-workspace/Megatron-LM/megatron/core/dist_checkpointiissing.ng/validation.py", line 542, in _validate_objects_for_key
[rank0]: raise CheckpointingException(err_msg) -Sep-25
[rank0]: megatron.core.dist_checkpointing.core.CheckpointingException: Invalid access pattern: 36 ShardedObject are missing.
Contributor guide
Assessment
This issue has not been assessed yet.