tensorflow / tensorflow/datasets
AssertionError: No examples were yielded for my local dataset based on DIV2K script
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
/!\ PLEASE INCLUDE THE FULL STACKTRACE AND CODE SNIPPET
Short description
I have a super resolution dataset which have lr and hr folder containing png images.
I follow the instruction on the site and based on div2k srcipt to write a load for my dataset.
My training cannot load the dataset with the tfds, and it tried to download some other dataset, i.e. I comment the download action.
Thanks for the help.
Environment information
-
Operating System: linux
-
Python version: <3.7>
-
tensorflow-datasets/tfds-nightlyversion: latest -
tensorflow/tf-nightlyversion: -
Does the issue still exists with the last
tfds-nightlypackage (pip install --upgrade tfds-nightly) ?
Reproduction instructions
class Grass(tfds.core.GeneratorBasedBuilder):
"""DatasetBuilder for face_grass dataset."""
VERSION = tfds.core.Version('1.0.0')
RELEASE_NOTES = {
'1.0.0': 'Initial release.',
}
def _info(self) -> tfds.core.DatasetInfo:
"""Returns the dataset metadata."""
# TODO(face_grass): Specifies the tfds.core.DatasetInfo object
return tfds.core.DatasetInfo(
builder=self,
description=_DESCRIPTION,
features=tfds.features.FeaturesDict({
"lr": tfds.features.Image(),
"hr": tfds.features.Image(),
}),
supervised_keys=("lr", "hr"),
# If there's a common (input, target) tuple from the
# features, specify them here. They'll be used if
# `as_supervised=True` in `builder.as_dataset`.
# supervised_keys=('image', 'label'), # Set to `None` to disable
homepage='https://dataset-homepage/',
citation=_CITATION,
)
def _split_generators(self, dl_manager: tfds.download.DownloadManager):
"""Returns SplitGenerators."""
# TODO(face_grass): Downloads the data and defines the splits
#path = dl_manager.download_and_extract('https://todo-data-url')
# TODO(face_grass): Returns the Dict[split names, Iterator[Key, Example]]
return [
tfds.core.SplitGenerator(
name=tfds.Split.TRAIN,
gen_kwargs={
"lr_path":
"../data1/project/grass_0808/train/LR",
"hr_path":
"../data1/project/grass_0808/train/HR",
}),
tfds.core.SplitGenerator(
name=tfds.Split.VALIDATION,
gen_kwargs={
"lr_path":
"../data1/project/grass_0808/valid/LR",
"hr_path":
"../data1/project/grass_0808/valid/HR",
}),
]
def _generate_examples(self, lr_path, hr_path):
"""Yields examples."""
# TODO(face_grass): Yields (key, example) tuples from the dataset
for root, _, files in tf.io.gfile.walk(lr_path):
for file_path in files:
# Select only png files.
if file_path.endswith(".png"):
yield file_path, {
"lr": os.path.join(root, file_path),
"hr": os.path.join(hr_path, file_path)
}
If you share a colab, make sure to update the permissions to share it.
Link to logs
error : Downloading and preparing dataset Unknown size (download: Unknown size, generated: Unknown size, total: Unknown size) to /root/tensorflow_datasets/grass/1.0.0...
Assertionerror: no examples were yielded.
Expected behavior
What you expected to happen.
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting the custom builder's _split_generators and _generate_examples methods in the issue, then run TFDS preparation with the local LR and HR paths. Verify that the paths are accessible and that the walk finds matching PNG files in both folders. Done means the builder yields examples and preparation completes without the no-examples assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100