tensorflow / tensorflow/datasets
Community Datasets not working on Windows system
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
After installing tfds-nightly from scratch, the community datasets didn't seem to work on windows as tfds.list_builders() did not display community datasets.
After checking the code, it seems that the fault lies in register_package.py.
While trying to create the community_dataset_index, TFDS initially needs to read the json file from the GCS_COMMUNITY_INDEX_PATH like this.
When it is not able to find the cached dataset index file on disk, TFDS tried to refresh the cache by accessing the json file from GCS
https://github.com/tensorflow/datasets/blob/d867411ea3f30971f4b532f68c48e48d1e4f30ff/tensorflow_datasets/core/community/register_package.py#L178
This is where the bug arises as _remote_path is PosixGPath('gs://tfds-data/community-datasets-list.jsonl' and TF currently does not support 'gs' file systems on Windows (mentioned here and here)
Due to this, community datasets do not seem to be working on Windows system (for tfds-nightly). However, they work perfectly fine with Linux OS and Colab. @Conchylicultor
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in tensorflow_datasets/core/community/register_package.py, especially the community dataset index cache read and refresh paths around the referenced lines. Reproduce the issue on Windows with a fresh tfds-nightly install and run tfds.list_builders(); done means community datasets are listed on Windows as they are on Linux and Colab.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100