facebookresearch / facebookresearch/fairscale

Error with nested models "Caffe2 uses a lazy allocation..."

Open
#1,129 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
293
PR merge metrics
No merged PRs in 30d

Description

I am trying to run a FSDP training built on top of metaseq:

I have a model with two other models nested inside it:

- FSDP(model);
- FSDP(submodel)
- FSDP(submodel);

However I get this error: ""RuntimeError: The tensor has a non-zero number of elements, but its data is not allocated yet. Caffe2 uses a lazy allocation, so you will need to call mutable_data() or raw_mutable"" during the forward pass of the first submodule, do you ever encountered this error? Thanks a lot!

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by reproducing the nested FSDP setup during the first submodule's forward pass and inspect the reported Caffe2 lazy-allocation error; done means the cause and a verified resolution are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.