google / google/riegeli

GCS support for FdReader used in (downstream) TFDS

Open
#31 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
495
Forks
66
PR merge metrics
No merged PRs in 30d

Description

Following up on [google/array_record/issues/120](https://github.com/google/array_record/issues/120)

The following code fails when it shouldn't

```python
import tensorflow_datasets as tfds
ds = tfds.data_source("ref_coco", data_dir="gs://ref_coco", try_gcs=True)
next(iter(ds['train']))
```

[Colab link](https://colab.research.google.com/drive/1iezVDZBJrWtP3fVTpAMNGb6qfENZNKuD)

The path gets handed over to a Riegeli `FdReader` ([source](https://github.com/google/array_record/blob/1f72e6d22336705df5079e2089e15a8208cdf351/python/array_record_module.cc#L94)) and the `array_record` maintainers have pointed me upstream. I can't find GCS support mentioned in the docs, but given TFDS provides a `try_gcs` argument, it seems like GCS buckets should be supported somewhere along the line.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.