huggingface / huggingface/accelerate

prepare_model unconditionally imports DTensor, breaking single-GPU training on torch builds without a distributed backend

Open
#4,249 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9.9k
Forks
1.5k
Avg merge
5d 2h
Merged PRs (30d)
27

Description

### What happens

`Accelerator.prepare_model` calls `model_has_dtensor(model)` while computing `device_placement`. That function imports `torch.distributed.tensor`, which transitively imports `torch._C._distributed_c10d`. On a PyTorch build compiled without a distributed backend, the import raises and every single-GPU run dies at trainer start.

AMD's official Windows ROCm wheels (`repo.amd.com/rocm/whl/...`) are such a build: `torch.distributed` imports, but `torch.distributed.is_available()` is `False` and `torch._C._distributed_c10d` does not exist. This currently breaks all training on AMD GPUs on Windows.

### Reproduction

```python
import torch, torch.nn as nn, accelerate
from accelerate import Accelerator

print("accelerate", accelerate.__version__)
print("has _distributed_c10d:", hasattr(torch._C, "_distributed_c10d"))
print("torch.distributed.is_available():", torch.distributed.is_available())

Accelerator().prepare_model(nn.Linear(4, 4))
print("PASS")
```

On `torch 2.11.0+rocm7.13.0` (Windows 11, Python 3.13), same machine, only the accelerate version differs:

```
accelerate 1.14.0
has _distributed_c10d: False
torch.distributed.is_available(): False
PASS

accelerate 1.15.0
has _distributed_c10d: False
torch.distributed.is_available(): False
ModuleNotFoundError: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package
```

Trimmed traceback on 1.15.0:

```
accelerate/accelerator.py:1802 in prepare_model -> and not model_has_dtensor(model)
accelerate/utils/other.py:243 in model_has_dtensor -> from torch.distributed.tensor import DTensor
torch/distributed/tensor/__init__.py:4
torch/distributed/tensor/_ops/_conv_ops.py:5
torch/distributed/tensor/_dtensor_spec.py:9
torch/distributed/tensor/placement_types.py:10
torch/distributed/_functional_collectives.py:9
torch/distributed/distributed_c10d.py:25 -> from torch._C._distributed_c10d import (...)
ModuleNotFoundError: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package
```

### Where it regressed

In v1.14.0 the only `model_has_dtensor(model)` call is at `accelerator.py:1878`, guarded by:

```python
if self.multi_device and not (self.parallelism_config and self.parallelism_config.tp_enabled):
```

so a single-GPU run never reached it.

v1.15.0 adds a second, unguarded call at `accelerator.py:1802`, inside the `device_placement` computation that every `prepare_model` executes. That appears to come from #4181 (`f7cc8e9a57993e049d21f2755fa85d3afc17af7b`, merged 2026-09-03), first released in 1.15.0 on 2026-09-09.

### Suggested fix

Either short-circuit `model_has_dtensor` when `torch.distributed.is_available()` is `False`, or restore the multi-device guard on the new call site. The first looks more robust, since a DTensor cannot be present at all without a distributed build.

Related, previously accepted as a supported configuration: #1787.

### Environment

- accelerate 1.15.0 (fails) / 1.14.0 (works)
- torch 2.11.0+rocm7.13.0, Windows 11, Python 3.13
- AMD Radeon AI PRO R9700 (gfx1201), ROCm 7.13
- Also reported by a user on a Radeon RX 6950 XT (gfx1030), Windows, stock install; downgrading to 1.14.0 restored training there as well

Contributor guide

Open the contributing guide

Research direction

Start in accelerate/accelerator.py at prepare_model and in accelerate/utils/other.py at model_has_dtensor. Run the provided reproduction with a PyTorch build where torch.distributed.is_available() is false, then check the existing guarded call and the new unguarded call. Done means single-GPU preparation no longer imports unavailable DTensor support while supported distributed configurations still work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.