tensorflow / tensorflow/datasets

tensorflow_datasets v4.9.4 introduces bug that prevents loading datasets

Open
#5,203 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Short description
When upgrading to the most recent tensorflow_datasets==4.9.4 I am getting errors for loading datasets (from the official TFDS catalogue). I have verified that the same datasets can load in version 4.9.3 without problem.

Environment information

Reproduction instructions

import tensorflow_datasets as tfds
ds = tfds.load("fractal20220817_data", data_dir="gs://gresearch/robotics")

OR colab: https://colab.research.google.com/drive/1neCJ3_TnF1tqr8qv4FM5__-v4SwVOOxJ?usp=sharing

Link to logs

FileNotFoundError                         Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/tensorflow_datasets/core/dataset_info.py](https://localhost:8080/#) in read_from_json(path)
   1033   try:
-> 1034     json_str = epath.Path(path).read_text()
   1035   except OSError as e:

27 frames
FileNotFoundError: [Errno 2] No such file or directory: 'fractal20220817_data/0.1.0/dataset_info.json'

The above exception was the direct cause of the following exception:

FileNotFoundError                         Traceback (most recent call last)
FileNotFoundError: Could not load dataset info from fractal20220817_data/0.1.0/dataset_info.json

The above exception was the direct cause of the following exception:

FileNotFoundError                         Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/tensorflow_datasets/core/utils/py_utils.py](https://localhost:8080/#) in reraise(e, prefix, suffix)
    383     else:
    384       exception = RuntimeError(f'{type(e).__name__}: {msg}')
--> 385     raise exception from e
    386   # Otherwise, modify the exception in-place
    387   elif len(e.args) <= 1:

FileNotFoundError: Failed to construct dataset "fractal20220817_data", builder_kwargs "{'data_dir': 'gs://gresearch/robotics'}": Could not load dataset info from fractal20220817_data/0.1.0/dataset_info.json

Additional context
Interestingly, constructing a builder_from_directory still seems to work even in the most recent tfds version.
builder = tfds.builder_from_directory("gs://gresearch/robotics/fractal20220817_data/0.1.0")

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Colab reproduction using tfds.load("fractal20220817_data", data_dir="gs://gresearch/robotics") and compare behavior between tensorflow-datasets 4.9.3 and 4.9.4. Trace the dataset-info loading path and compare it with builder_from_directory, which still works; done means the official dataset loads successfully from the documented data directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.