google / google/grain

random and augmentation in datasets

Open
#1,102 1 comment 0 reactions 0 assignees View on GitHub
type:support
Dominant language
Python
Stars
779
Forks
86
Avg merge
2d 6h
Merged PRs (30d)
6

Description

I am trying to do an on-fly-augmentation using the dataset.
```python
train_dataset = (grain.MapDataset.source(source)
.map(augment())
.shuffle(seed=412)
.repeat(num_epochs=config.n_epochs)
.batch(batch_size=config.batch_size, drop_remainder=True)
)
train_dataset = grain.experimental.ThreadPrefetchIterDataset(train_dataset, prefetch_buffer_size=64)
train_dataset = train_dataset.map(jax.device_put)
```
The augment() requires random seeds. But the train_dataset will be prefeched in parallel threads, and I am not sure how to pass the random seed. Any suggestions?

ps. I am also not sure that the shuffle was done correctly. Please correct me if I'm wrong.

Thanks a lot!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.