Lightning-AI / Lightning-AI/lightning-thunder
DTensor: Get it working on the thunderfx path when requires_grad=True.
@kshitij12345 is already working on this.
Since Jun 13, 2025.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
This currently doesn't work.
```python
from torch.distributed.tensor import DTensor
from torch.distributed import init_device_mesh
from thunder.dynamo import thunderfx
import torch
import os
# Create mesh
os.environ["RANK"] = "0"
os.environ["LOCAL_RANK"] = "0"
os.environ["WORLD_SIZE"] = "1"
os.environ["MASTER_ADDR"] = "localhost"
os.environ["MASTER_PORT"] = "29500"
mesh = init_device_mesh("cuda", (1,), mesh_dim_names=["i"])
hidden_size = 16
def model(x, w):
return torch.mul(x, w)
weight = DTensor.from_local(torch.randn(hidden_size, hidden_size, requires_grad=True), device_mesh=mesh)
in_dtensor = DTensor.from_local(torch.randn(hidden_size, hidden_size, requires_grad=True), device_mesh=mesh)
expected = torch.compile(model)(in_dtensor, weight)
tmodel = thunderfx(model)
actual = tmodel(in_dtensor, weight)
torch.testing.assert_close(actual.to_local(), expected.to_local())
try:
assert tmodel._backend.subgraph_infos[0].split_reasons == [] # Fails
finally:
torch.distributed.destroy_process_group()
```
Split reasons:
```py
In [2]: tmodel._backend.subgraph_infos[0].split_reasons
Out[2]: [SplitReason(reason_type=, info='Failed while running meta for node with name: mul and target: , see exception field', exception='aten.mul.Tensor: got mixed torch.Tensor and DTensor, need to convert all torch.Tensor to DTensor before calling distributed operators!')]
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.