huggingface / huggingface/accelerate

WebLoader Support

Open
#2,083 9 comments 0 reactions 0 assignees View on GitHub
enhancement feature request
Dominant language
Python
Stars
9.9k
Forks
1.5k
Avg merge
5d 2h
Merged PRs (30d)
27

Description

### System Info

```Shell
CUDA 11.8
My environment is:
- accelerate==0.24.0
- braceexpand==0.1.7
- certifi==2022.12.7
- charset-normalizer==2.1.1
- diffusers==0.21.4
- filelock==3.9.0
- fsspec==2023.4.0
- huggingface-hub==0.17.3
- idna==3.4
- imageio==2.31.6
- importlib-metadata==6.8.0
- jinja2==3.1.2
- lazy-loader==0.3
- markupsafe==2.1.2
- mpmath==1.3.0
- natsort==8.4.0
- networkx==3.0
- numpy==1.24.1
- opencv-python==4.8.1.78
- packaging==23.2
- pandas==2.1.1
- pillow==9.3.0
- psutil==5.9.6
- python-dateutil==2.8.2
- pytz==2023.3.post1
- pyyaml==6.0.1
- regex==2023.10.3
- requests==2.28.1
- safetensors==0.4.0
- scikit-image==0.22.0
- scipy==1.11.3
- six==1.16.0
- sympy==1.12
- tifffile==2023.9.26
- tokenizers==0.14.1
- torch==2.1.0+cu118
- torchaudio==2.1.0+cu118
- torchvision==0.16.0+cu118
- tqdm==4.66.1
- transformers==4.34.1
- triton==2.1.0
- typing-extensions==4.4.0
- tzdata==2023.3
- urllib3==1.26.13
- webdataset==0.2.62
- zipp==3.17.0
```

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [X] My own task or dataset (give details below)

### Reproduction

I am trying to use webdataset and webloader in my training code.I can run my code without any error if i am using webdataset and dataloader of torch,however if i want to use webdataset and webloader,i am getting this error:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:1! (when checking argument for argument weight in method wrapper_CUDA___slow_conv2d_forward)

### Expected behavior

Works without any error as webdataset+dataloader configuration

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.