Lightning-AI / Lightning-AI/lightning-thunder

[distributed] Enable transformed modules to load state dicts of the originals

Open
#564 2 comments 0 reactions 2 assignees Claimed by @t-vi View on GitHub
distributed enhancement tensor parallel transforms
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature

Design and implement a helper to load checkpoints (in the sense of `state_dict`s of modules) from pre-transformed onto transformed modules. An example snippet would look as follows:

```python
import os

import torch.nn
from thunder.distributed import column_parallel
from thunder.distributed import row_parallel

class Model(nn.Module):
...

local_rank = int(os.environ["LOCAL_RANK"])
device = torch.device(f"cuda:{local_rank}")
c10d.init_process_group()

model = Model(...).to(device)
jitted_model = thunder.jit(model, ...)
tensor_parallel_model = row_parallel(column_parallel(jitted_model, ....), ...)
orig_state_dict = model.state_dict() # or jitted_model.state_dict()
tensor_parallel_model.load_state_dict(orig_state_dict)
```

### Motivation

For transforms that could change the shape of parameters, checkpoint loading and creating a checkpoint would not be simple as ones free from param modifications.
For example, fsdp and tensor parallel transforms would need such feature.
When generating a checkpoint, they need to unshard parameters. When loading a checkpoint, they would need to shard parameters appropriately.

cc @carmocca @crcrpar

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.