Lightning-AI / Lightning-AI/lightning-thunder
[distributed] Enable transformed modules to load state dicts of the originals
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
## 🚀 Feature
Design and implement a helper to load checkpoints (in the sense of `state_dict`s of modules) from pre-transformed onto transformed modules. An example snippet would look as follows:
```python
import os
import torch.nn
from thunder.distributed import column_parallel
from thunder.distributed import row_parallel
class Model(nn.Module):
...
local_rank = int(os.environ["LOCAL_RANK"])
device = torch.device(f"cuda:{local_rank}")
c10d.init_process_group()
model = Model(...).to(device)
jitted_model = thunder.jit(model, ...)
tensor_parallel_model = row_parallel(column_parallel(jitted_model, ....), ...)
orig_state_dict = model.state_dict() # or jitted_model.state_dict()
tensor_parallel_model.load_state_dict(orig_state_dict)
```
### Motivation
For transforms that could change the shape of parameters, checkpoint loading and creating a checkpoint would not be simple as ones free from param modifications.
For example, fsdp and tensor parallel transforms would need such feature.
When generating a checkpoint, they need to unshard parameters. When loading a checkpoint, they would need to shard parameters appropriately.
cc @carmocca @crcrpar
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.