google-deepmind / google-deepmind/open_x_embodiment
Heavy load on the gs://gresearch/robotics bucket
- Dominant language
- Jupyter Notebook
- Stars
- 2k
- Forks
- 127
- PR merge metrics
- No merged PRs in 30d
Description
First thanks for the awesome work of unifying the datasets!
Im currently trying to train a model using datasets from the gs://gresearch/robotics repo, which worked very good for a long time but since the last few days everything kind of slowed down.
Is there currently a heavy load on the google Server or is this a problem on my end?
I tried 2 different Clusters and both showed bad results for tfds.benchmark:
```python
import tensorflow as tf
import tensorflow_datasets as tfds
tf.config.set_visible_devices([], "GPU")
builder = tfds.builder_from_directory(builder_dir="gs://gresearch/robotics/droid/1.0.0")
ds = builder.as_dataset(split="train[:95%]")
ds = ds.shuffle(1000) # not needed but important for training so i added it
ds = ds.batch(32).prefetch(buffer_size=tf.data.AUTOTUNE)
tfds.benchmark(ds, batch_size=256)
it = iter(ds)
for i in range(2):
episode = next(it)
# steps = list(episode['steps'])
print("done")
_________________________
Examples/sec (First included) 182.94 ex/sec (total: 701440 ex, 3834.31 sec)
Examples/sec (First only) 2.71 ex/sec (total: 256 ex, 94.32 sec)
Examples/sec (First excluded) 187.48 ex/sec (total: 701184 ex, 3739.99 sec)
```
And also bad results for "Computing dataset statistics" using the Octo Dataloader for pytorch (from ~200it/s some weeks ago to ~1.66s/it)
Just want to make sure if it is a problem on my end (e.g. change in TF versions / Net connection) or if currently many people load data and it is only heavy load.
Contributor guide
Assessment
This issue has not been assessed yet.