tensorflow / tensorflow/datasets
Error while loading Dementiabank dataset
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Short description
There's an error when loading the Dementiabank dataset, although manually downloaded files are in the correct directory.
Environment information
-
Operating System: Ubuntu 18.04
-
Python version: 3.8.10
-
tensorflow-datasets/tfds-nightlyversion: 4.3.0 -
tensorflow/tf-nightlyversion: 2.4.1 -
Does the issue still exists with the last
tfds-nightlypackage (pip install --upgrade tfds-nightly) ? - Yes
Reproduction instructions
import tensorflow_datasets as tfds
tfds.load('Dementiabank')
Link to logs
Downloading and preparing dataset Unknown size (download: Unknown size, generated: 17.71 GiB, total: 17.71 GiB) to /home/admin/tensorflow_datasets/dementiabank/1.0.0...
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
~/projects/voice/trill/foo.py in
27 import tensorflow_datasets as tfds
28
----> 29 tfds.load('Dementiabank', batch_size=-1)
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/load.py in load(name, split, data_dir, batch_size, shuffle_files, download, as_supervised, decoders, read_config, with_info, builder_kwargs, download_and_prepare_kwargs, as_dataset_kwargs, try_gcs)
331 if with_info:
332 return ds, dbuilder.info
--> 333 return ds
334
335
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py in download_and_prepare(self, download_dir, download_config)
437 # Old version of TF are not os.PathLike compatible
438 with tf_compat.mock_gfile_pathlike():
--> 439 self._download_and_prepare(
440 dl_manager=dl_manager,
441 download_config=download_config,
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py in _download_and_prepare(self, dl_manager, download_config)
1153 self.info.file_format].FILE_SUFFIX
1154
-> 1155 split_info_futures = [
1156 split_builder.submit_split_generation( # pylint: disable=g-complex-comprehension
1157 split_name=split_name,
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py in <listcomp>(.0)
1154
1155 split_info_futures = [
-> 1156 split_builder.submit_split_generation( # pylint: disable=g-complex-comprehension
1157 split_name=split_name,
1158 generator=generator,
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/split_builder.py in submit_split_generation(self, split_name, generator, path, disable_shuffling)
289 # `_build_from_xyz` method.
290 if isinstance(generator, collections.abc.Iterable):
--> 291 return self._build_from_generator(**build_kwargs)
292 else: # Otherwise, beam required
293 unknown_generator_type = TypeError(
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/split_builder.py in _build_from_generator(self, split_name, generator, path, disable_shuffling)
359 example = self._features.encode_example(example)
360 except Exception as e: # pylint: disable=broad-except
--> 361 utils.reraise(e, prefix=f'Failed to encode example:\n{example}\n')
362 writer.write(key, example)
363 shard_lengths, total_size = writer.finalize()
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/split_builder.py in _build_from_generator(self, split_name, generator, path, disable_shuffling)
357 ):
358 try:
--> 359 example = self._features.encode_example(example)
360 except Exception as e: # pylint: disable=broad-except
361 utils.reraise(e, prefix=f'Failed to encode example:\n{example}\n')
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/features/features_dict.py in encode_example(self, example_dict)
196 utils.reraise(
197 e, prefix=f'In <{feature.__class__.__name__}> with name "{k}":\n')
--> 198 return example
199
200 def _flatten(self, x):
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/features/features_dict.py in encode_example(self, example_dict)
194 example[k] = feature.encode_example(example_value)
195 except Exception as e: # pylint: disable=broad-except
--> 196 utils.reraise(
197 e, prefix=f'In <{feature.__class__.__name__}> with name "{k}":\n')
198 return example
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/features/audio_feature.py in encode_example(self, audio_or_path_or_fobj)
95 else:
96 return self._encode_file(audio_or_path_or_fobj, self._file_format)
---> 97
98 @property
99 def sample_rate(self):
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/features/audio_feature.py in encode_example(self, audio_or_path_or_fobj)
93 except Exception as e: # pylint: disable=broad-except
94 utils.reraise(e, prefix=f'Error for {filename}: ')
---> 95 else:
96 return self._encode_file(audio_or_path_or_fobj, self._file_format)
97
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/features/audio_feature.py in _encode_file(self, fobj, file_format)
83
84 def encode_example(self, audio_or_path_or_fobj):
---> 85 if isinstance(audio_or_path_or_fobj, (np.ndarray, list)):
86 return audio_or_path_or_fobj
87 elif isinstance(audio_or_path_or_fobj, type_utils.PathLikeCls):
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/features/feature.py in encode_example(self, example_data)
697
698 def encode_example(self, example_data):
--> 699 """See base class for details."""
700 np_dtype = np.dtype(self.dtype.as_numpy_dtype)
701 if isinstance(example_data, tf.Tensor):
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/utils/tf_utils.py in assert_shape_match(shape1, shape2)
143 Args:
144 shape1 (tuple): Static shape
--> 145 shape2 (tuple): Dynamic shape (can contain None)
146 """
147 shape1 = tf.TensorShape(shape1)
~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py in assert_same_rank(self, other)
999 if self.rank is not None and other.rank is not None:
1000 if self.rank != other.rank:
-> 1001 raise ValueError("Shapes %s and %s must have the same rank" %
1002 (self, other))
1003
ValueError: Failed to encode example:
{'audio': '/home/admin/tensorflow_datasets/downloads/manual/dementia/English/Pitt/Control/cookie/006-2.mp3', 'label': 'control', 'speaker_id': '006'}
In <Audio> with name "audio":
Error for /home/admin/tensorflow_datasets/downloads/manual/dementia/English/Pitt/Control/cookie/006-2.mp3: Shapes (1922688, 2) and (None,) must have the same rank
Expected behavior
Dataset loads.
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the provided tfds.load('Dementiabank') reproduction and the Audio encoder shown in core/features/audio_feature.py, then inspect how the listed MP3 is handled. Reproduce the rank mismatch with the stated environment and confirm completion when the Dementiabank dataset loads successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100