huggingface / huggingface/accelerate

all_gather_into_tensor is deprecated, _gpu_gather warns on every gather

Open Beginner friendly
#4,269 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9.9k
Forks
1.5k
Avg merge
5d 2h
Merged PRs (30d)
27

Description

### System Info

accelerate 1.15.0, torch 2.13.0+cu129, transformers 5.18.0.dev0, 8 x H100.

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Reproduction

`_gpu_gather` uses an API torch deprecated, so every `Accelerator.gather` logs a FutureWarning once per rank:

```
torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated.
Please use `torch.distributed.all_gather_single` instead.
```

```python
# utils/operations.py:324
gather_op = torch.distributed.all_gather_into_tensor
```

It is reached from any Trainer run, `Trainer._get_num_items_in_batch` -> `Accelerator.gather` -> `_gpu_gather` -> `_gpu_gather_one`.

### Expected behavior

No warning. `torch.distributed.all_gather_single` exists in torch 2.13, so this can be a rename behind a version check for older torch.

Contributor guide

Open the contributing guide

Research direction

Start in utils/operations.py around line 324 and trace _gpu_gather through _gpu_gather_one, then check the reported torch 2.13 API and older-version compatibility. The work is done when Accelerator.gather no longer emits the deprecation warning while remaining compatible with supported torch versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.