tensorflow / tensorflow/datasets

One folder per dataset

Open
#2,139 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Currently, we're grouping files per file type: (checksums/, fake_data/, datasets/).
We should try to move toward a more modular approach where datasets are self-contained directories:

dataset_type0/   # image, text, etc
    dataset0/
        dataset0.py
        dataset0_test.py
        checksums.tsv
        dummy_data/
    dataset1/
        dataset1.py
        ...

TODOs:

  1. We should make it easy for external users to follow this model:
    Checksums path, fake data,... should be automatically detected without the user having to do the additional setup (tfds.core.add_checksums_dir, FAKE_DATA_DIR,...)

  2. Update scripts like create_new_dataset, document_dataset, etc.

  3. Update dataset testing methods

  4. Update all files depending on checksums, fake_dataset locations etc

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the current checksums/, fake_data/, and datasets/ layout, then inspect the create_new_dataset and document_dataset scripts and dataset testing methods. Map every dependency on checksum and fake-data locations before changing the organization. Done means datasets are self-contained, paths are detected automatically, scripts and tests support the new model, and all dependent files are updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.