huggingface / huggingface/accelerate

Not possible to use the notebook_launcher on a cluster of A6000 series

Open
#721 22 comments 0 reactions 1 assignee Claimed by @muellerzr View on GitHub
bug feature request
Dominant language
Python
Stars
9.9k
Forks
1.5k
Avg merge
5d 2h
Merged PRs (30d)
27

Description

### System Info

```Shell
Copy-and-paste the text below in your GitHub issue

- `Accelerate` version: 0.12.0
- Platform: Linux-5.4.0-105-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.13
- Numpy version: 1.21.5
- PyTorch version (GPU?): 1.12.0 (True)
- `Accelerate` default config:
- compute_environment: LOCAL_MACHINE
- distributed_type: MULTI_GPU
- mixed_precision: bf16
- use_cpu: False
- num_processes: 8
- machine_rank: 0
- num_machines: 1
- main_process_ip: None
- main_process_port: None
- main_training_function: main
- deepspeed_config: {}
- fsdp_config: {}
- downcast_bf16: False
```

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [X] My own task or dataset (give details below)

### Reproduction

```
def training_loop(mixed_precision="bf16", seed: int = 42, batch_size: int = 64):
set_seed(seed)

accelerator = Accelerator(mixed_precision=mixed_precision)

args = ("bf16", 42, 64)
notebook_launcher(training_loop, args, num_processes=8)

```

### Expected behavior

I expect training to start with autocasting to bfloat16 - instead, I get the error: "RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method"

AFAICT, this is due to the "is_bf16_available" function in accelerate.utils, which calls "torch.cuda.is_available()" and "torch.cuda.is_bf16_supported()".

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.