tensorflow / tensorflow/datasets

Speech Commands Dataset: Test set different from the Set given in testing_list.txt

Open
#3,061 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

What I need help with / What I was wondering
For the Google Speech Commands Dataset, it seems to be common practice to derive the test set from the file testing_list.txt found in the speech_commands_v0.02.tar.gz archive. This file contains 11005 filenames, drawn from the original distribution of 20 words with ca. 400 samples each and no separate class for background noise.

Different from that, Tensorflow Datasets generates the test set from an archive that is downloaded separately:
https://github.com/tensorflow/datasets/blob/34aeabf177a0723d14a902e9cc9022e7a16ab6fc/tensorflow_datasets/audio/speech_commands.py#L54

Inspecting the test split generated by TFDS shows that it contains 4890 test samples with around 400 samples for the 10 target words along with an equal amount of samples for the background noise and unknown classes. The original testing list seems to be parsed and excluded from the training set, but isn't used otherwise
https://github.com/tensorflow/datasets/blob/34aeabf177a0723d14a902e9cc9022e7a16ab6fc/tensorflow_datasets/audio/speech_commands.py#L166-L168

This means that the test set used is significantly different from the training/validation sets because it's not skewed towards the unknown class. This bias is also present in the test set generated from the testing_list.txt, but not here. I understand how these balanced classes are more useful for assessing model performance, however it makes the performance also much harder to compare to the numbers found in literature.

Is there any other reasoning behind using the different test set?
Is there a simple way of changing the default behavior to use the samples given in the testing_list file?

It would be nice if...
There was a hint regarding the difference in test sets on the description page. Ideally even a switch that allows a user to select which one they would want to use.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tensorflow_datasets/audio/speech_commands.py, especially the referenced lines that download the alternate archive and exclude testing_list.txt entries. Compare the generated splits with the archive's testing_list.txt and review the Speech Commands dataset description page. Done means the test-set difference is documented and, if feasible, users can select the desired split source.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.