huggingface / huggingface/datasets
Support buffer shuffle for to_tf_dataset
Open
enhancement
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Feature request
I'm using to_tf_dataset to convert a large dataset to tf.data.Dataset and use Keras fit to train model.
Currently, to_tf_dataset only supports full size shuffle, which can be very slow on large dataset.
tf.data.Dataset support buffer shuffle by default.
shuffle(
buffer_size, seed=None, reshuffle_each_iteration=None, name=None
)
### Motivation
I'm very frustrated to find the loading with shuffling large dataset is very slow. It seems impossible to shuffle before training Keras with big dataset.
### Your contribution
NA
Contributor guide
Assessment
This issue has not been assessed yet.