Lightning-AI / Lightning-AI/pytorch-lightning
TransformerEnginePrecision _convert_layers(module) fails for FSDP zero2/zero3
Open
Nobody has claimed this yet.
bug
precision: te
strategy: fsdp
ver: 2.2.x
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
Bug description
TransformerEnginePrecision.convert_module function seems to not work for the the FSDP-wrapped model.
What version are you seeing the problem on?
master
How to reproduce the bug
model = FSDP(
model,
sharding_strategy=sharding_strategy,
auto_wrap_policy=custom_wrap_policy,
device_id=local_rank,
use_orig_params=True,
device_mesh=mesh,
)
te_precision = TransformerEnginePrecision(weights_dtype=torch.bfloat16, replace_layers=True)
self.model = te_precision.convert_module(self.model)
Error messages and logs
[rank1]: self.model = te_precision.convert_module(self.model)
[rank1]: _convert_layers(module)
[rank1]: File "/usr/local/lib/python3.10/dist-packages/lightning/fabric/plugins/precision/transformer_engine.py", line 165, in _convert_layers
[rank1]: replacement.weight.data = child.weight.data.clone()
[rank1]: RuntimeError: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.
More info
I actually see it for pytorch-lightning==2.3.0
cc @lantiga
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with lightning/fabric/plugins/precision/transformer_engine.py, especially _convert_layers and TransformerEnginePrecision.convert_module. Reproduce the failure using an FSDP-wrapped model with zero2 or zero3 sharding and the shown precision settings. Done means conversion completes without the incompatible tensor type error for the reported FSDP configurations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100