Lightning-AI / Lightning-AI/pytorch-lightning

TransformerEnginePrecision _convert_layers(module) fails for FSDP zero2/zero3

Open
#19,989 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug precision: te strategy: fsdp ver: 2.2.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

Bug description

TransformerEnginePrecision.convert_module function seems to not work for the the FSDP-wrapped model.

What version are you seeing the problem on?

master

How to reproduce the bug
model = FSDP(
    model,
    sharding_strategy=sharding_strategy,
    auto_wrap_policy=custom_wrap_policy,
    device_id=local_rank,
    use_orig_params=True,
    device_mesh=mesh,
)
te_precision = TransformerEnginePrecision(weights_dtype=torch.bfloat16, replace_layers=True)
self.model = te_precision.convert_module(self.model)
Error messages and logs
[rank1]:     self.model = te_precision.convert_module(self.model)
[rank1]:     _convert_layers(module)
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/lightning/fabric/plugins/precision/transformer_engine.py", line 165, in _convert_layers
[rank1]:     replacement.weight.data = child.weight.data.clone()
[rank1]: RuntimeError: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.
More info

I actually see it for pytorch-lightning==2.3.0

cc @lantiga

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with lightning/fabric/plugins/precision/transformer_engine.py, especially _convert_layers and TransformerEnginePrecision.convert_module. Reproduce the failure using an FSDP-wrapped model with zero2 or zero3 sharding and the shown precision settings. Done means conversion completes without the incompatible tensor type error for the reported FSDP configurations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.