google-deepmind / google-deepmind/open_x_embodiment

Heavy load on the gs://gresearch/robotics bucket

Open
#79 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2k
Forks
127
PR merge metrics
No merged PRs in 30d

Description

First thanks for the awesome work of unifying the datasets!

Im currently trying to train a model using datasets from the gs://gresearch/robotics repo, which worked very good for a long time but since the last few days everything kind of slowed down.
Is there currently a heavy load on the google Server or is this a problem on my end?
I tried 2 different Clusters and both showed bad results for tfds.benchmark:

```python
import tensorflow as tf
import tensorflow_datasets as tfds

tf.config.set_visible_devices([], "GPU")
builder = tfds.builder_from_directory(builder_dir="gs://gresearch/robotics/droid/1.0.0")
ds = builder.as_dataset(split="train[:95%]")
ds = ds.shuffle(1000) # not needed but important for training so i added it
ds = ds.batch(32).prefetch(buffer_size=tf.data.AUTOTUNE)
tfds.benchmark(ds, batch_size=256)

it = iter(ds)
for i in range(2):
episode = next(it)

# steps = list(episode['steps'])
print("done")
_________________________

Examples/sec (First included) 182.94 ex/sec (total: 701440 ex, 3834.31 sec)
Examples/sec (First only) 2.71 ex/sec (total: 256 ex, 94.32 sec)
Examples/sec (First excluded) 187.48 ex/sec (total: 701184 ex, 3739.99 sec)
```

And also bad results for "Computing dataset statistics" using the Octo Dataloader for pytorch (from ~200it/s some weeks ago to ~1.66s/it)

Just want to make sure if it is a problem on my end (e.g. change in TF versions / Net connection) or if currently many people load data and it is only heavy load.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.