tensorflow / tensorflow/datasets
How to properly import and load custom datasets
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
What I need help with / What I was wondering
I cannot import my custom tfds dataset that does not require downloading, and the section in the documentation on how to import it is not clear to me.
What I've tried so far
I ran the following code in a notebook:
import data.rfcx_tfds
builder = tfds.builder('rfcx_tfds')
builder.as_dataset()
Which gives the following error (also, ds = tfds.load('rfcx_tfds', download=False) gives the same error):
AssertionError: Dataset rfcx_tfds: could not find data in /home/maximus/tensorflow_datasets. Please make sure to call dataset_builder.download_and_prepare(), or pass download=True to tfds.load() before trying to access the tf.data.Dataset object.
In order to rectify this, I put "import data.rfcx_tfds" in __init__.py files that live in the root dir, the notebook dir, and the data/rfcx_tfds dir (it was already there). It is also run in the cell of the notebook when I try to load the dataset. I still cannot load my dataset, even though I think I've done what was asked of me in the documentation.
It would be nice if...
I don't understand what part of this section of the documentation that I am doing incorrectly. I am able to make the builder object, but nothing past that. Besides, I'd rather call tfds.load than tfds.builder and the .as_dataset().
My final goal is to load data directly from .wav files into tf tensors, without having to download anything. I could not find any example of loading a custom dataset that uses locally stored data, so I'm still not sure if this issue is because I haven't stored data in tfrecords or because I'm improperly importing my dataset.
Environment information
(if applicable)
- Operating System: Ubuntu 18.04
- Python version: 3.7.6
tensorflow-datasetsversion: 4.0.1tensorflowversion: 2.3.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked “Add an import for registration” documentation section and reproduce the notebook calls to tfds.builder, builder.as_dataset(), and tfds.load(download=False). Clarify how a locally stored custom dataset, including .wav files, should be imported and loaded without downloading, and document whether TFRecord preparation is required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- data, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100