huggingface / huggingface/accelerate
IterableDataset with batch samples of different size
Open
feature request
- Dominant language
- Python
- Stars
- 9.9k
- Forks
- 1.5k
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 27
Description
Hi,
I want to train a large dataset with accelerate.
However, my dataset has different shape (same batch, different sequence length) of samples.
This [line](https://github.com/huggingface/accelerate/blob/b2956acbe925845bf202224797334cfe4da18396/src/accelerate/data_loader.py#L356) requires the big batch for different processes to maintain a same shape so that it can concat.
I was wondering if we can just broadcast the tensors to other processors without forcing the data to be of same shape?
Contributor guide
Assessment
This issue has not been assessed yet.