huggingface / huggingface/accelerate
XLA + Gradient Accumulation does not work
- Dominant language
- Python
- Stars
- 9.9k
- Forks
- 1.5k
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 27
Description
### System Info
```Shell
- `Accelerate` version: 0.11.0.dev0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Numpy version: 1.21.6
- PyTorch version (GPU?): 1.9.0+cu102 (False)
- `Accelerate` default config:
- compute_environment: LOCAL_MACHINE
- distributed_type: TPU
- mixed_precision: no
- use_cpu: False
- num_processes: 8
- machine_rank: 0
- num_machines: 1
- main_process_ip: None
- main_process_port: None
- main_training_function: main
- deepspeed_config: {}
- fsdp_config: {}
```
### Information
- [X] The official example scripts
- [ ] My own modified scripts
### Tasks
- [X] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [ ] My own task or dataset (give details below)
### Reproduction
1. Run [this](https://raw.githubusercontent.com/huggingface/accelerate/main/examples/nlp_example.py) version of the simple_nlp script and change it's batch size to be one greater than `MAX_BATCH_SIZE_GPU`
2. See results, and they perform poorly
### Expected behavior
```Shell
Running on gradient accumulation vs a very large batch size should hold no difference.
```
Contributor guide
Assessment
This issue has not been assessed yet.