huggingface / huggingface/accelerate
prepare_model unconditionally imports DTensor, breaking single-GPU training on torch builds without a distributed backend
- Dominant language
- Python
- Stars
- 9.9k
- Forks
- 1.5k
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 27
Description
### What happens
`Accelerator.prepare_model` calls `model_has_dtensor(model)` while computing `device_placement`. That function imports `torch.distributed.tensor`, which transitively imports `torch._C._distributed_c10d`. On a PyTorch build compiled without a distributed backend, the import raises and every single-GPU run dies at trainer start.
AMD's official Windows ROCm wheels (`repo.amd.com/rocm/whl/...`) are such a build: `torch.distributed` imports, but `torch.distributed.is_available()` is `False` and `torch._C._distributed_c10d` does not exist. This currently breaks all training on AMD GPUs on Windows.
### Reproduction
```python
import torch, torch.nn as nn, accelerate
from accelerate import Accelerator
print("accelerate", accelerate.__version__)
print("has _distributed_c10d:", hasattr(torch._C, "_distributed_c10d"))
print("torch.distributed.is_available():", torch.distributed.is_available())
Accelerator().prepare_model(nn.Linear(4, 4))
print("PASS")
```
On `torch 2.11.0+rocm7.13.0` (Windows 11, Python 3.13), same machine, only the accelerate version differs:
```
accelerate 1.14.0
has _distributed_c10d: False
torch.distributed.is_available(): False
PASS
accelerate 1.15.0
has _distributed_c10d: False
torch.distributed.is_available(): False
ModuleNotFoundError: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package
```
Trimmed traceback on 1.15.0:
```
accelerate/accelerator.py:1802 in prepare_model -> and not model_has_dtensor(model)
accelerate/utils/other.py:243 in model_has_dtensor -> from torch.distributed.tensor import DTensor
torch/distributed/tensor/__init__.py:4
torch/distributed/tensor/_ops/_conv_ops.py:5
torch/distributed/tensor/_dtensor_spec.py:9
torch/distributed/tensor/placement_types.py:10
torch/distributed/_functional_collectives.py:9
torch/distributed/distributed_c10d.py:25 -> from torch._C._distributed_c10d import (...)
ModuleNotFoundError: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package
```
### Where it regressed
In v1.14.0 the only `model_has_dtensor(model)` call is at `accelerator.py:1878`, guarded by:
```python
if self.multi_device and not (self.parallelism_config and self.parallelism_config.tp_enabled):
```
so a single-GPU run never reached it.
v1.15.0 adds a second, unguarded call at `accelerator.py:1802`, inside the `device_placement` computation that every `prepare_model` executes. That appears to come from #4181 (`f7cc8e9a57993e049d21f2755fa85d3afc17af7b`, merged 2026-09-03), first released in 1.15.0 on 2026-09-09.
### Suggested fix
Either short-circuit `model_has_dtensor` when `torch.distributed.is_available()` is `False`, or restore the multi-device guard on the new call site. The first looks more robust, since a DTensor cannot be present at all without a distributed build.
Related, previously accepted as a supported configuration: #1787.
### Environment
- accelerate 1.15.0 (fails) / 1.14.0 (works)
- torch 2.11.0+rocm7.13.0, Windows 11, Python 3.13
- AMD Radeon AI PRO R9700 (gfx1201), ROCm 7.13
- Also reported by a user on a Radeon RX 6950 XT (gfx1030), Windows, stock install; downgrading to 1.14.0 restored training there as well
Contributor guide
Research direction
Start in accelerate/accelerator.py at prepare_model and in accelerate/utils/other.py at model_has_dtensor. Run the provided reproduction with a PyTorch build where torch.distributed.is_available() is false, then check the existing guarded call and the new unguarded call. Done means single-GPU preparation no longer imports unavailable DTensor support while supported distributed configurations still work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100