tensorflow / tensorflow/datasets

How to properly import and load custom datasets

Open
#2,680 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

What I need help with / What I was wondering
I cannot import my custom tfds dataset that does not require downloading, and the section in the documentation on how to import it is not clear to me.

What I've tried so far
I ran the following code in a notebook:

import data.rfcx_tfds
builder = tfds.builder('rfcx_tfds')
builder.as_dataset()

Which gives the following error (also, ds = tfds.load('rfcx_tfds', download=False) gives the same error):

AssertionError: Dataset rfcx_tfds: could not find data in /home/maximus/tensorflow_datasets. Please make sure to call dataset_builder.download_and_prepare(), or pass download=True to tfds.load() before trying to access the tf.data.Dataset object.

In order to rectify this, I put "import data.rfcx_tfds" in __init__.py files that live in the root dir, the notebook dir, and the data/rfcx_tfds dir (it was already there). It is also run in the cell of the notebook when I try to load the dataset. I still cannot load my dataset, even though I think I've done what was asked of me in the documentation.

It would be nice if...
I don't understand what part of this section of the documentation that I am doing incorrectly. I am able to make the builder object, but nothing past that. Besides, I'd rather call tfds.load than tfds.builder and the .as_dataset().

My final goal is to load data directly from .wav files into tf tensors, without having to download anything. I could not find any example of loading a custom dataset that uses locally stored data, so I'm still not sure if this issue is because I haven't stored data in tfrecords or because I'm improperly importing my dataset.

Environment information
(if applicable)

  • Operating System: Ubuntu 18.04
  • Python version: 3.7.6
  • tensorflow-datasets version: 4.0.1
  • tensorflow version: 2.3.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked “Add an import for registration” documentation section and reproduce the notebook calls to tfds.builder, builder.as_dataset(), and tfds.load(download=False). Clarify how a locally stored custom dataset, including .wav files, should be imported and loaded without downloading, and document whether TFRecord preparation is required.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.