facebookresearch / facebookresearch/fairscale
[FSDP] Wrapping model again in FSDP doesn't contain root parameters
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 293
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
Related https://github.com/PyTorchLightning/pytorch-lightning/pull/6152
When wrapping the module twice in FSDP, because we introduce a `FlattenParamsWrapper` that contains all the parameters, this means the second wrapping does not contain the parameters for the model. This is required for Lightning where we wrap the `LightningModule` in training, and return this `LightningModule` back to the user who may call `trainer.test(module)` independently.
This potentially could be solved by removing `FlattenParamsWrapper` and returning sharded weights back to the correct references permanently after training. Is this doable?
I'm not 100% on a solution here (even from the Lightning side) so if you have ideas please let me know!
## To Reproduce
```python
import os
import unittest
from unittest import mock
import torch
import torch.nn as nn
from fairscale.nn import FullyShardedDataParallel
import torch.nn.functional as F
@mock.patch.dict(os.environ, {"MASTER_ADDR": "localhost", "MASTER_PORT": "1337"}, clear=True)
@unittest.skipIf(not torch.cuda.is_available(), "Test Requires CUDA")
def test_wrapping_module():
"""
This test simulates wrapping the module after training to run inference.
This is required in cases where later in a session, the model is wrapped again in FSDP but
contains nested FSDP wrappers within the module.
"""
device = torch.device("cuda")
torch.cuda.set_device(0)
torch.distributed.init_process_group(backend="nccl", rank=0, world_size=1)
module = nn.Sequential(
nn.Linear(5, 5),
FullyShardedDataParallel(nn.Linear(5, 5)),
)
training_model = FullyShardedDataParallel(module).to(device)
input = torch.rand((1, 5), dtype=torch.float).to(device)
output = training_model(input)
loss = F.mse_loss(input, output)
loss.backward()
inference_model = FullyShardedDataParallel(module).to(device)
second_output = inference_model(input)
assert torch.allclose(output, second_output)
# Fails as we are missing parameters in the highest root level FSDP wrap
assert len(list(inference_model.parameters())) == len(list(training_model.parameters()))
torch.distributed.destroy_process_group()
```
## Expected behavior
Able to wrap the model in an FSDP wrapper again after model is trained.
cc @ananthsub @min-xu-ai @shuyingsunshine21
Contributor guide
Assessment
This issue has not been assessed yet.