tensorflow / tensorflow/datasets

Problem loading and converting to tfrecords for quickdraw_bitmap dataset

Open
#2,440 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

/!\ IF YOU WANT PEOPLE TO HELP YOU, PLEASE GIVE AS MUCH DETAIL AS POSSIBLE, INCLUDING THE FULL STACKTRACE AND CODE SNIPPET

Short description
Cannot load quickdraw_bitmap dataset and convert into tfrecords.

Environment information

  • Operating System: Ubuntu 20.04
  • Python version: Python 3.8.2
  • tensorflow-datasets/tfds-nightly version: 3.2.1
  • tensorflow/tensorflow-gpu/tf-nightly/tf-nightly-gpu version: 2.3.0

Reproduction instructions

import tensorflow_datasets as tfds
builder = tfds.builder('quickdraw_bitmap', data_dir='./')
builder.download_and_prepare()

Link to logs
Traceback (most recent call last):
File "", line 1, in
File "/home/peidaqi/.local/lib/python3.8/site-packages/tensorflow_datasets/core/api_utils.py", line 69, in disallow_positional_args_dec
return fn(*args, **kwargs)
File "/home/peidaqi/.local/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py", line 374, in download_and_prepare
self._download_and_prepare(
File "/home/peidaqi/.local/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py", line 1017, in _download_and_prepare
super(GeneratorBasedBuilder, self)._download_and_prepare(
File "/home/peidaqi/.local/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py", line 951, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File "/home/peidaqi/.local/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py", line 1033, in _prepare_split
for key, record in utils.tqdm(generator, unit=" examples",
File "/home/peidaqi/.local/lib/python3.8/site-packages/tqdm/std.py", line 1129, in iter
for obj in iterable:
File "/home/peidaqi/.local/lib/python3.8/site-packages/tensorflow_datasets/image_classification/quickdraw.py", line 113, in _generate_examples
class_images = np.load(f)
File "/home/peidaqi/.local/lib/python3.8/site-packages/numpy/lib/npyio.py", line 452, in load
return format.read_array(fid, allow_pickle=allow_pickle,
File "/home/peidaqi/.local/lib/python3.8/site-packages/numpy/lib/format.py", line 777, in read_array
data = _read_bytes(fp, read_size, "array data")
File "/home/peidaqi/.local/lib/python3.8/site-packages/numpy/lib/format.py", line 914, in _read_bytes
raise ValueError(msg % (error_template, size, len(data)))
ValueError: EOF: reading array data, expected 262144 bytes got 262064

EOF: reading array data, expected 262144 bytes got 262064

Expected behavior
What you expected to happen.

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tensorflow_datasets/image_classification/quickdraw.py at _generate_examples, where the traceback reaches np.load, and run the reported builder.download_and_prepare() reproduction. Check the download and preparation path for quickdraw_bitmap; done means the dataset converts to TFRecords without the reported EOF error.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, tensorflow
Domain
data, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.