tensorflow / tensorflow/datasets

tfds errors with tfds.testing.mock_data

Open
#3,637 5 comments 0 reactions 1 assignee View on GitHub

@tomvdw is already working on this.

Since Dec 28, 2021.

help
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

What I need help with / What I was wondering
How to use tfds.testing.mock_data correctly

What I've tried so far
This code on its own works fine:

import tensorflow_datasets as tfds
ds_builder = tfds.builder('mnist')
ds_builder.download_and_prepare()
train_ds = tfds.as_numpy(ds_builder.as_dataset(split='train', batch_size=-1))

When decorating that snippet with tfds.testing.mock_data I get errors:

with tfds.testing.mock_data(num_examples=8, data_dir='/workdisk/.tfds2/metadata'):
  import tensorflow_datasets as tfds
  ds_builder = tfds.builder('mnist')
  ds_builder.download_and_prepare()
  train_ds = tfds.as_numpy(ds_builder.as_dataset(split='train', batch_size=-1))

Error trace:

I1220 13:56:57.120099 140470378489664 mocking.py:162] Metadata NOT found for mnist at /workdisk/jax/FlaxExamples/flax/.tfds/metadata/mnist/3.0.1. Will use `MockPolicy.USE_CODE.`
I1220 13:56:57.121221 140470378489664 logging_logger.py:35] Constructing tf.data.Dataset mnist for split train, from /workdisk/jax/FlaxExamples/flax/.tfds/metadata/mnist/3.0.1
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow_datasets/core/dataset_builder.py:622: get_single_element (from tensorflow.python.data.experimental.ops.get_single_element) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.get_single_element()`.
W1220 13:56:57.204224 140470378489664 deprecation.py:341] From /usr/local/lib/python3.8/dist-packages/tensorflow_datasets/core/dataset_builder.py:622: get_single_element (from tensorflow.python.data.experimental.ops.get_single_element) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.get_single_element()`.
terminate called after throwing an instance of 'std::length_error'
  what():  vector::reserve
Fatal Python error: Aborted

Environment information
Latest Nvidia dev docker image: nvidia/cuda:11.4.2-cudnn8-devel-ubuntu20.04

  • Operating System: ubuntu20.04
  • Python version: 3.8
  • tensorflow 2.7.0
  • tensorflow-datasets 4.4.0
  • tensorflow-estimator 2.7.0
  • tensorflow-io-gcs-filesystem 0.23.1
  • tensorflow-metadata 1.5.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.