tensorflow / tensorflow/datasets

gs' not implemented

Open
#5,324 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

/!\ PLEASE INCLUDE THE FULL STACKTRACE AND CODE SNIPPET

Short description
Description of the bug.

Environment information

  • Operating System: windows
  • Python version: 3.11.8
  • tensorflow-datasets version: 4.9.4
  • tensorflow version: 2.15.0

Reproduction instructions

git clone https://github.com/tensorflow/datasets.git
cd datasets
pip install -e .
python -c "import tensorflow_datasets as tfds; tfds = tfds.load('berkeley_autolab_ur5')"

Link to logs

Traceback (most recent call last):
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\utils\py_utils.py", line 420, in try_reraise
    yield
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\load.py", line 222, in builder
    return cls(**builder_kwargs)  # pytype: disable=not-instantiable
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\logging\__init__.py", line 288, in decorator
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\dataset_builder.py", line 1362, in __init__
    super().__init__(**kwargs)
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\logging\__init__.py", line 288, in decorator
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\dataset_builder.py", line 285, in __init__
    self.info.initialize_from_bucket()
    ^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\logging\__init__.py", line 168, in __call__
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\dataset_builder.py", line 474, in info
    info = self._info()
           ^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\robotics\dataset_importer_builder.py", line 82, in _info
    features = self.get_ds_builder().info.features
               ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\robotics\dataset_importer_builder.py", line 150, in get_ds_builder    
    ds_builder = tfds.builder_from_directory(ds_location)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\read_only_builder.py", line 149, in builder_from_directory       
    return ReadOnlyBuilder(builder_dir=builder_dir)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\logging\__init__.py", line 288, in decorator
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\read_only_builder.py", line 65, in __init__
    info_proto = dataset_info.read_proto_from_builder_dir(builder_dir)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\dataset_info.py", line 1059, in read_proto_from_builder_dir      
    return read_from_json(info_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\dataset_info.py", line 1035, in read_from_json
    json_str = epath.Path(path).read_text()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\etils\epath\abstract_path.py", line 157, in read_text
    return f.read()
           ^^^^^^^^
  File "C:\Users\Zelong\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\lib\io\file_io.py", line 116, in read
    self._preread_check()
  File "C:\Users\Zelong\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\lib\io\file_io.py", line 77, in _preread_check
    self._read_buf = _pywrap_file_io.BufferedInputStream(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 'gs' not implemented (file: 'gs://gresearch/robotics/berkeley_autolab_ur5/0.1.0/dataset_info.json')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\logging\__init__.py", line 168, in __call__
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\load.py", line 643, in load
    dbuilder = _fetch_builder(
               ^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\load.py", line 498, in _fetch_builder
    return builder(name, data_dir=data_dir, try_gcs=try_gcs, **builder_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2288.0_x64__qbz5n2kfra8p0\Lib\contextlib.py", line 81, in inner   
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\logging\__init__.py", line 168, in __call__
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\load.py", line 219, in builder
    with py_utils.try_reraise(
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2288.0_x64__qbz5n2kfra8p0\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\utils\py_utils.py", line 422, in try_reraise
    reraise(e, *args, **kwargs)
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\utils\py_utils.py", line 389, in reraise
    raise exception from e
RuntimeError: UnimplementedError: Failed to construct dataset "berkeley_autolab_ur5", builder_kwargs "{'data_dir': None}": File system scheme 'gs' not implemented (file: 'gs://gresearch/robotics/berkeley_autolab_ur5/0.1.0/dataset_info.json')

Expected behavior
It should find and download the dataset

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported command for berkeley_autolab_ur5 and trace the failure through tensorflow_datasets/core/load.py, robotics/dataset_importer_builder.py, core/read_only_builder.py, and core/dataset_info.py. Compare the gs:// path handling with the TensorFlow file I/O call shown in the stack trace. Done means the dataset can be found and downloaded from the reported Windows environment without the gs filesystem error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.