tensorflow / tensorflow/datasets

Error while loading Dementiabank dataset

Open
#3,322 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Short description
There's an error when loading the Dementiabank dataset, although manually downloaded files are in the correct directory.

Environment information

  • Operating System: Ubuntu 18.04

  • Python version: 3.8.10

  • tensorflow-datasets/tfds-nightly version: 4.3.0

  • tensorflow/tf-nightly version: 2.4.1

  • Does the issue still exists with the last tfds-nightly package (pip install --upgrade tfds-nightly) ? - Yes

Reproduction instructions

import tensorflow_datasets as tfds

tfds.load('Dementiabank')

Link to logs

Downloading and preparing dataset Unknown size (download: Unknown size, generated: 17.71 GiB, total: 17.71 GiB) to /home/admin/tensorflow_datasets/dementiabank/1.0.0...
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/projects/voice/trill/foo.py in 
      27 import tensorflow_datasets as tfds
      28 
----> 29 tfds.load('Dementiabank', batch_size=-1)

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/load.py in load(name, split, data_dir, batch_size, shuffle_files, download, as_supervised, decoders, read_config, with_info, builder_kwargs, download_and_prepare_kwargs, as_dataset_kwargs, try_gcs)
    331   if with_info:
    332     return ds, dbuilder.info
--> 333   return ds
    334 
    335 

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py in download_and_prepare(self, download_dir, download_config)
    437           # Old version of TF are not os.PathLike compatible
    438           with tf_compat.mock_gfile_pathlike():
--> 439             self._download_and_prepare(
    440                 dl_manager=dl_manager,
    441                 download_config=download_config,

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py in _download_and_prepare(self, dl_manager, download_config)
   1153           self.info.file_format].FILE_SUFFIX
   1154 
-> 1155       split_info_futures = [
   1156           split_builder.submit_split_generation(  # pylint: disable=g-complex-comprehension
   1157               split_name=split_name,

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py in <listcomp>(.0)
   1154 
   1155       split_info_futures = [
-> 1156           split_builder.submit_split_generation(  # pylint: disable=g-complex-comprehension
   1157               split_name=split_name,
   1158               generator=generator,

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/split_builder.py in submit_split_generation(self, split_name, generator, path, disable_shuffling)
    289     # `_build_from_xyz` method.
    290     if isinstance(generator, collections.abc.Iterable):
--> 291       return self._build_from_generator(**build_kwargs)
    292     else:  # Otherwise, beam required
    293       unknown_generator_type = TypeError(

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/split_builder.py in _build_from_generator(self, split_name, generator, path, disable_shuffling)
    359         example = self._features.encode_example(example)
    360       except Exception as e:  # pylint: disable=broad-except
--> 361         utils.reraise(e, prefix=f'Failed to encode example:\n{example}\n')
    362       writer.write(key, example)
    363     shard_lengths, total_size = writer.finalize()

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/split_builder.py in _build_from_generator(self, split_name, generator, path, disable_shuffling)
    357     ):
    358       try:
--> 359         example = self._features.encode_example(example)
    360       except Exception as e:  # pylint: disable=broad-except
    361         utils.reraise(e, prefix=f'Failed to encode example:\n{example}\n')

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/features/features_dict.py in encode_example(self, example_dict)
    196         utils.reraise(
    197             e, prefix=f'In <{feature.__class__.__name__}> with name "{k}":\n')
--> 198     return example
    199 
    200   def _flatten(self, x):

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/features/features_dict.py in encode_example(self, example_dict)
    194         example[k] = feature.encode_example(example_value)
    195       except Exception as e:  # pylint: disable=broad-except
--> 196         utils.reraise(
    197             e, prefix=f'In <{feature.__class__.__name__}> with name "{k}":\n')
    198     return example

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/features/audio_feature.py in encode_example(self, audio_or_path_or_fobj)
     95     else:
     96       return self._encode_file(audio_or_path_or_fobj, self._file_format)
---> 97 
     98   @property
     99   def sample_rate(self):

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/features/audio_feature.py in encode_example(self, audio_or_path_or_fobj)
     93         except Exception as e:  # pylint: disable=broad-except
     94           utils.reraise(e, prefix=f'Error for {filename}: ')
---> 95     else:
     96       return self._encode_file(audio_or_path_or_fobj, self._file_format)
     97 

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/features/audio_feature.py in _encode_file(self, fobj, file_format)
     83 
     84   def encode_example(self, audio_or_path_or_fobj):
---> 85     if isinstance(audio_or_path_or_fobj, (np.ndarray, list)):
     86       return audio_or_path_or_fobj
     87     elif isinstance(audio_or_path_or_fobj, type_utils.PathLikeCls):

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/features/feature.py in encode_example(self, example_data)
    697 
    698   def encode_example(self, example_data):
--> 699     """See base class for details."""
    700     np_dtype = np.dtype(self.dtype.as_numpy_dtype)
    701     if isinstance(example_data, tf.Tensor):

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow_datasets/core/utils/tf_utils.py in assert_shape_match(shape1, shape2)
    143   Args:
    144     shape1 (tuple): Static shape
--> 145     shape2 (tuple): Dynamic shape (can contain None)
    146   """
    147   shape1 = tf.TensorShape(shape1)

~/.miniconda3/envs/tf/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py in assert_same_rank(self, other)
    999     if self.rank is not None and other.rank is not None:
   1000       if self.rank != other.rank:
-> 1001         raise ValueError("Shapes %s and %s must have the same rank" %
   1002                          (self, other))
   1003 

ValueError: Failed to encode example:
{'audio': '/home/admin/tensorflow_datasets/downloads/manual/dementia/English/Pitt/Control/cookie/006-2.mp3', 'label': 'control', 'speaker_id': '006'}
In <Audio> with name "audio":
Error for /home/admin/tensorflow_datasets/downloads/manual/dementia/English/Pitt/Control/cookie/006-2.mp3: Shapes (1922688, 2) and (None,) must have the same rank

Expected behavior
Dataset loads.

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the provided tfds.load('Dementiabank') reproduction and the Audio encoder shown in core/features/audio_feature.py, then inspect how the listed MP3 is handled. Reproduce the rank mismatch with the stated environment and confirm completion when the Dementiabank dataset loads successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.