google / google/grain

Performance gap with `tf.data.Dataset` on MNIST

Open
#1,164 5 comments 0 reactions 0 assignees View on GitHub
type:performance
Dominant language
Python
Stars
779
Forks
86
Avg merge
2d 6h
Merged PRs (30d)
6

Description

I'm experiencing a serious performance gap between `tf.data.Dataset` using TFRecord and `grain` w/ ArrayRecord. Simply trying to iterate over MNIST with comparable speed. Snippet below exemplifies, Grain seems 5-10x slower.

Trying to make comparison fair, so I've disabled caching and prefetch for TF. I've tried adding a variety of Grain functionality to improve speed, including `mp_prefetch`, varying `num_threads`, using `CacheIterDataset` and `ThreadPrefetchIterDataset`, etc.

FYI, I've also found one TF option that slows the performance down to Grain-level: `num_parallel_calls_for_decode` in `tfds.ReadConfig`.

```python
import time
from pathlib import Path

import grain.python as grain
import tensorflow as tf
import tensorflow_datasets as tfds
import tqdm

tf.config.set_visible_devices([], device_type="GPU")

n_epoch = 10
batch_size = 32

def make_tf_dataset():
ds = tfds.load(
"mnist",
split="train",
data_dir=Path.home() / "datasets/tfrecord",
read_config=tfds.ReadConfig(
options=tf.data.Options(), try_autocache=False, skip_prefetch=True
),
)

ds = ds.repeat(n_epoch)
ds = ds.batch(batch_size)
ds = tfds.as_numpy(ds)

return ds

def make_grain_dataset():
source = tfds.data_source(
"mnist", split="train", data_dir=Path.home() / "datasets/array_record"
)
ds = grain.MapDataset.source(source)

ds = ds.repeat(n_epoch)
ds = ds.batch(batch_size)
ds = ds.to_iter_dataset(grain.ReadOptions(num_threads=8))

return ds

def profile(ds):
t0 = time.perf_counter()
for _ in tqdm.tqdm(iter(ds)):
pass
t1 = time.perf_counter()
return t1 - t0

if __name__ == "__main__":
t_tf = profile(make_tf_dataset())
print(f"TF: {t_tf:.3f} s")

t_grain = profile(make_grain_dataset())
print(f"Grain: {t_grain:.3f} s")
```

I keep trying to make Grain work for me every few months (since #569), but with no avail, and it seems like other performance issues are going largely unresolved. I love JAX and want to switch to Grain, but trying to get it to work is making me ☹️

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.