tensorflow / tensorflow/datasets
Problem loading and converting to tfrecords for quickdraw_bitmap dataset
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
/!\ IF YOU WANT PEOPLE TO HELP YOU, PLEASE GIVE AS MUCH DETAIL AS POSSIBLE, INCLUDING THE FULL STACKTRACE AND CODE SNIPPET
Short description
Cannot load quickdraw_bitmap dataset and convert into tfrecords.
Environment information
- Operating System: Ubuntu 20.04
- Python version: Python 3.8.2
tensorflow-datasets/tfds-nightlyversion: 3.2.1tensorflow/tensorflow-gpu/tf-nightly/tf-nightly-gpuversion: 2.3.0
Reproduction instructions
import tensorflow_datasets as tfds
builder = tfds.builder('quickdraw_bitmap', data_dir='./')
builder.download_and_prepare()
Link to logs
Traceback (most recent call last):
File "", line 1, in
File "/home/peidaqi/.local/lib/python3.8/site-packages/tensorflow_datasets/core/api_utils.py", line 69, in disallow_positional_args_dec
return fn(*args, **kwargs)
File "/home/peidaqi/.local/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py", line 374, in download_and_prepare
self._download_and_prepare(
File "/home/peidaqi/.local/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py", line 1017, in _download_and_prepare
super(GeneratorBasedBuilder, self)._download_and_prepare(
File "/home/peidaqi/.local/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py", line 951, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File "/home/peidaqi/.local/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py", line 1033, in _prepare_split
for key, record in utils.tqdm(generator, unit=" examples",
File "/home/peidaqi/.local/lib/python3.8/site-packages/tqdm/std.py", line 1129, in iter
for obj in iterable:
File "/home/peidaqi/.local/lib/python3.8/site-packages/tensorflow_datasets/image_classification/quickdraw.py", line 113, in _generate_examples
class_images = np.load(f)
File "/home/peidaqi/.local/lib/python3.8/site-packages/numpy/lib/npyio.py", line 452, in load
return format.read_array(fid, allow_pickle=allow_pickle,
File "/home/peidaqi/.local/lib/python3.8/site-packages/numpy/lib/format.py", line 777, in read_array
data = _read_bytes(fp, read_size, "array data")
File "/home/peidaqi/.local/lib/python3.8/site-packages/numpy/lib/format.py", line 914, in _read_bytes
raise ValueError(msg % (error_template, size, len(data)))
ValueError: EOF: reading array data, expected 262144 bytes got 262064
EOF: reading array data, expected 262144 bytes got 262064
Expected behavior
What you expected to happen.
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tensorflow_datasets/image_classification/quickdraw.py at _generate_examples, where the traceback reaches np.load, and run the reported builder.download_and_prepare() reproduction. Check the download and preparation path for quickdraw_bitmap; done means the dataset converts to TFRecords without the reported EOF error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python, tensorflow
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100