huggingface / huggingface/accelerate

IterableDataset with batch samples of different size

Open
#348 3 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Python
Stars
9.9k
Forks
1.5k
Avg merge
5d 2h
Merged PRs (30d)
27

Description

Hi,
I want to train a large dataset with accelerate.
However, my dataset has different shape (same batch, different sequence length) of samples.
This [line](https://github.com/huggingface/accelerate/blob/b2956acbe925845bf202224797334cfe4da18396/src/accelerate/data_loader.py#L356) requires the big batch for different processes to maintain a same shape so that it can concat.

I was wondering if we can just broadcast the tensors to other processors without forcing the data to be of same shape?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.