tensorflow / tensorflow/datasets

Multi-Config Dataset Loading / Wildcard Config Identifiers

Open
#2,900 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Some datasets, e.g. mnist_corrupted provide various configurations which are in many use-cases all used at the same time (without distinction). Afaik, tfds currently requires to load every dataset independenty and then concatenate them, e.g.

c1_ds = tfds.load("mnist_corrupted/shot_noise")
c2_ds = tfds.load("mnist_corrupted/impulse_noise")
c3_ds = tfds.load("mnist_corrupted/glass_blur")
# ... many, many, many rows :-)
cX_ds = tfds.load("mnist_corrupted/...")

dataset_i_want = some_concat_function((c1_ds,c2_ds, c3_ds, ..., cX_ds))

For tfds with a large number of configs, this snippet can become quite long. Also, it makes it hard to use some of the nice features of load over the full dataset (e.g. shuffle_files or split).

Describe the solution you'd like
Some way to use wildcards over tfds configs, e.g.,

# Contains the datasets for all configs, nicely shuffled
dataset_i_want = tfds.load("mnist_corrupted/*", shuffle_files=True)

or

# Contains the shot_noise and impulse_noise config datasets
dataset_i_want = tfds.load("mnist_corrupted/*_noise", shuffle_files=True)

Describe alternatives you've considered

Option 1: Manual concatenations, as shown in the example above.
Option 2: Looping over all BuilderConfigs in the DatasetBuilder class (e.g. MNISTCorrupted.BUILDER_CONFIGS), manually implementing wildcard pattern matching on the config names. However, I do not think the BUILDER_CONFIGS field is documented and probably not guaranteed to exist on all DatasetBuilders? Also, shuffling and splitting is still hard.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the tfds.load entry point and the DatasetBuilder/BUILDER_CONFIGS behavior described in the issue. Determine how wildcard config identifiers could select multiple configurations while preserving shuffle_files and split behavior. Done means a documented wildcard form such as mnist_corrupted/* loads the matching configurations as one dataset with those options applied.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
api, data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.