huggingface / huggingface/accelerate

XLA + Gradient Accumulation does not work

Open
#480 0 comments 0 reactions 1 assignee Claimed by @muellerzr View on GitHub
bug feature request TPU
Dominant language
Python
Stars
9.9k
Forks
1.5k
Avg merge
5d 2h
Merged PRs (30d)
27

Description

### System Info

```Shell
- `Accelerate` version: 0.11.0.dev0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Numpy version: 1.21.6
- PyTorch version (GPU?): 1.9.0+cu102 (False)
- `Accelerate` default config:
- compute_environment: LOCAL_MACHINE
- distributed_type: TPU
- mixed_precision: no
- use_cpu: False
- num_processes: 8
- machine_rank: 0
- num_machines: 1
- main_process_ip: None
- main_process_port: None
- main_training_function: main
- deepspeed_config: {}
- fsdp_config: {}
```

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Run [this](https://raw.githubusercontent.com/huggingface/accelerate/main/examples/nlp_example.py) version of the simple_nlp script and change it's batch size to be one greater than `MAX_BATCH_SIZE_GPU`
2. See results, and they perform poorly

### Expected behavior

```Shell
Running on gradient accumulation vs a very large batch size should hold no difference.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.