NVIDIA / NVIDIA/Megatron-LM

RuntimeError: Trying to resize storage that is not resizable in fine-grained activation offloading when force-releasing MoE expert_fc1 input that is a CUDA-graph static output

Open
#7,009 0 comments 0 reactions 1 assignee Claimed by @wujingyue View on GitHub
bug community-request waiting-on-maintainers
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

Description:
When fine-grained activation offloading is combined with TransformerEngine partial CUDA graphs on a MoE model (DeepSeek V3 lite), training crashes on the first forward right after CUDA graph capture:
**RuntimeError: Trying to resize storage that is not resizable**

Location: megatron/core/pipeline_parallel/fine_grained_activation_offload.py, bulk_offload():
**release_tensor.record_stream(cur_stream)
release_tensor.untyped_storage().resize_(0) # throws here**

Environment:
Megatron-LM: Release0.19
TransformerEngine: 2.14.1
torch: 2.10
Hardware: A100*8

Reproduction:
--cuda-graph-impl transformer_engine
--cuda-graph-scope moe_router moe_preprocess attn
--optimizer-cuda-graph
--fine-grained-activation-offloading
--offload-modules core_attn attn_proj expert_fc1

Fails on the first training step, immediately after "Start CUDA Graphs capture... / Time spent in CUDA Graphs capture".

Stack trace (rank0)
experts.py:825 fc1_output = expert_fc1_manager.group_offload(
fc1_output, forced_released_tensors=[permuted_local_hidden_states], ...)
fine_grained_activation_offload.py:1288 fine_grained_offloading_group_offload(...)
fine_grained_activation_offload.py:1235 FineGrainedOffloadingGroupCommitFunction.forward
fine_grained_activation_offload.py:1129 on_group_commit_forward(name, forced_released_tensors)
fine_grained_activation_offload.py:1106 release_tensor.untyped_storage().resize_(0)
RuntimeError: Trying to resize storage that is not resizable

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.