huggingface / huggingface/accelerate
webdataset compatibility / example
Open
documentation
wip
- Dominant language
- Python
- Stars
- 9.9k
- Forks
- 1.5k
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 27
Description
[webdataset](https://github.com/webdataset/webdataset) is a common system for handling datasets, which supports sharded and distributed training. I'm curious if it is easy to use with accelerate distributed training, or if there are anticipated problems. An example would be helpful to include in the documentation.
Contributor guide
Assessment
This issue has not been assessed yet.