tensorflow / tensorflow/datasets
Multi-Config Dataset Loading / Wildcard Config Identifiers
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Some datasets, e.g. mnist_corrupted provide various configurations which are in many use-cases all used at the same time (without distinction). Afaik, tfds currently requires to load every dataset independenty and then concatenate them, e.g.
c1_ds = tfds.load("mnist_corrupted/shot_noise")
c2_ds = tfds.load("mnist_corrupted/impulse_noise")
c3_ds = tfds.load("mnist_corrupted/glass_blur")
# ... many, many, many rows :-)
cX_ds = tfds.load("mnist_corrupted/...")
dataset_i_want = some_concat_function((c1_ds,c2_ds, c3_ds, ..., cX_ds))
For tfds with a large number of configs, this snippet can become quite long. Also, it makes it hard to use some of the nice features of load over the full dataset (e.g. shuffle_files or split).
Describe the solution you'd like
Some way to use wildcards over tfds configs, e.g.,
# Contains the datasets for all configs, nicely shuffled
dataset_i_want = tfds.load("mnist_corrupted/*", shuffle_files=True)
or
# Contains the shot_noise and impulse_noise config datasets
dataset_i_want = tfds.load("mnist_corrupted/*_noise", shuffle_files=True)
Describe alternatives you've considered
Option 1: Manual concatenations, as shown in the example above.
Option 2: Looping over all BuilderConfigs in the DatasetBuilder class (e.g. MNISTCorrupted.BUILDER_CONFIGS), manually implementing wildcard pattern matching on the config names. However, I do not think the BUILDER_CONFIGS field is documented and probably not guaranteed to exist on all DatasetBuilders? Also, shuffling and splitting is still hard.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the tfds.load entry point and the DatasetBuilder/BUILDER_CONFIGS behavior described in the issue. Determine how wildcard config identifiers could select multiple configurations while preserving shuffle_files and split behavior. Done means a documented wildcard form such as mnist_corrupted/* loads the matching configurations as one dataset with those options applied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- api, data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100