RuntimeError: Trying to resize storage that is not resizable in fine-grained activation offloading when force-releasing MoE expert_fc1 input that is a CUDA-graph static output
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
Description:
When fine-grained activation offloading is combined with TransformerEngine partial CUDA graphs on a MoE model (DeepSeek V3 lite), training crashes on the first forward right after CUDA graph capture:
**RuntimeError: Trying to resize storage that is not resizable**
Location: megatron/core/pipeline_parallel/fine_grained_activation_offload.py, bulk_offload():
**release_tensor.record_stream(cur_stream)
release_tensor.untyped_storage().resize_(0) # throws here**
Environment:
Megatron-LM: Release0.19
TransformerEngine: 2.14.1
torch: 2.10
Hardware: A100*8
Reproduction:
--cuda-graph-impl transformer_engine
--cuda-graph-scope moe_router moe_preprocess attn
--optimizer-cuda-graph
--fine-grained-activation-offloading
--offload-modules core_attn attn_proj expert_fc1
Fails on the first training step, immediately after "Start CUDA Graphs capture... / Time spent in CUDA Graphs capture".
Stack trace (rank0)
experts.py:825 fc1_output = expert_fc1_manager.group_offload(
fc1_output, forced_released_tensors=[permuted_local_hidden_states], ...)
fine_grained_activation_offload.py:1288 fine_grained_offloading_group_offload(...)
fine_grained_activation_offload.py:1235 FineGrainedOffloadingGroupCommitFunction.forward
fine_grained_activation_offload.py:1129 on_group_commit_forward(name, forced_released_tensors)
fine_grained_activation_offload.py:1106 release_tensor.untyped_storage().resize_(0)
RuntimeError: Trying to resize storage that is not resizable
Contributor guide
Assessment
This issue has not been assessed yet.