tensorflow / tensorflow/datasets

Community Datasets not working on Windows system

Open
#3,109 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

After installing tfds-nightly from scratch, the community datasets didn't seem to work on windows as tfds.list_builders() did not display community datasets.
After checking the code, it seems that the fault lies in register_package.py.
While trying to create the community_dataset_index, TFDS initially needs to read the json file from the GCS_COMMUNITY_INDEX_PATH like this.

When it is not able to find the cached dataset index file on disk, TFDS tried to refresh the cache by accessing the json file from GCS
https://github.com/tensorflow/datasets/blob/d867411ea3f30971f4b532f68c48e48d1e4f30ff/tensorflow_datasets/core/community/register_package.py#L178
This is where the bug arises as _remote_path is PosixGPath('gs://tfds-data/community-datasets-list.jsonl' and TF currently does not support 'gs' file systems on Windows (mentioned here and here)

Due to this, community datasets do not seem to be working on Windows system (for tfds-nightly). However, they work perfectly fine with Linux OS and Colab. @Conchylicultor

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tensorflow_datasets/core/community/register_package.py, especially the community dataset index cache read and refresh paths around the referenced lines. Reproduce the issue on Windows with a fresh tfds-nightly install and run tfds.list_builders(); done means community datasets are listed on Windows as they are on Linux and Colab.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.