facebookresearch / facebookresearch/fairscale
Error with nested models "Caffe2 uses a lazy allocation..."
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 293
- PR merge metrics
- No merged PRs in 30d
Description
I am trying to run a FSDP training built on top of metaseq:
I have a model with two other models nested inside it:
- FSDP(model);
- FSDP(submodel)
- FSDP(submodel);
However I get this error: ""RuntimeError: The tensor has a non-zero number of elements, but its data is not allocated yet. Caffe2 uses a lazy allocation, so you will need to call mutable_data() or raw_mutable"" during the forward pass of the first submodule, do you ever encountered this error? Thanks a lot!
Contributor guide
Research direction
The issue names no files or tests. Start by reproducing the nested FSDP setup during the first submodule's forward pass and inspect the reported Caffe2 lazy-allocation error; done means the cause and a verified resolution are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100