tensorflow / tensorflow/datasets

AssertionError: No examples were yielded for my local dataset based on DIV2K script

Open
#3,786 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

/!\ PLEASE INCLUDE THE FULL STACKTRACE AND CODE SNIPPET

Short description
I have a super resolution dataset which have lr and hr folder containing png images.
I follow the instruction on the site and based on div2k srcipt to write a load for my dataset.
My training cannot load the dataset with the tfds, and it tried to download some other dataset, i.e. I comment the download action.
Thanks for the help.

Environment information

  • Operating System: linux

  • Python version: <3.7>

  • tensorflow-datasets/tfds-nightly version: latest

  • tensorflow/tf-nightly version:

  • Does the issue still exists with the last tfds-nightly package (pip install --upgrade tfds-nightly) ?

Reproduction instructions

class Grass(tfds.core.GeneratorBasedBuilder):
  """DatasetBuilder for face_grass dataset."""

  VERSION = tfds.core.Version('1.0.0')
  RELEASE_NOTES = {
      '1.0.0': 'Initial release.',
  }

  def _info(self) -> tfds.core.DatasetInfo:
    """Returns the dataset metadata."""
    # TODO(face_grass): Specifies the tfds.core.DatasetInfo object
    return tfds.core.DatasetInfo(
        builder=self,
        description=_DESCRIPTION,
        features=tfds.features.FeaturesDict({
            "lr": tfds.features.Image(),
            "hr": tfds.features.Image(),
        }),
        supervised_keys=("lr", "hr"),
        # If there's a common (input, target) tuple from the
        # features, specify them here. They'll be used if
        # `as_supervised=True` in `builder.as_dataset`.
        # supervised_keys=('image', 'label'),  # Set to `None` to disable
        homepage='https://dataset-homepage/',
        citation=_CITATION,
    )

  def _split_generators(self, dl_manager: tfds.download.DownloadManager):
    """Returns SplitGenerators."""
    # TODO(face_grass): Downloads the data and defines the splits
    #path = dl_manager.download_and_extract('https://todo-data-url')

    # TODO(face_grass): Returns the Dict[split names, Iterator[Key, Example]]
    return [
        tfds.core.SplitGenerator(
            name=tfds.Split.TRAIN,
            gen_kwargs={
                "lr_path":
                    "../data1/project/grass_0808/train/LR",
                "hr_path":
                    "../data1/project/grass_0808/train/HR",
            }),
        tfds.core.SplitGenerator(
            name=tfds.Split.VALIDATION,
            gen_kwargs={
                "lr_path":
                    "../data1/project/grass_0808/valid/LR",
                "hr_path":
                    "../data1/project/grass_0808/valid/HR",
            }),
    ]

  def _generate_examples(self, lr_path, hr_path):
    """Yields examples."""
    # TODO(face_grass): Yields (key, example) tuples from the dataset
    for root, _, files in tf.io.gfile.walk(lr_path):
      for file_path in files:
        # Select only png files.
        if file_path.endswith(".png"):
          yield file_path, {
              "lr": os.path.join(root, file_path),
            
              "hr": os.path.join(hr_path, file_path)
          }

If you share a colab, make sure to update the permissions to share it.

Link to logs
error : Downloading and preparing dataset Unknown size (download: Unknown size, generated: Unknown size, total: Unknown size) to /root/tensorflow_datasets/grass/1.0.0...

Assertionerror: no examples were yielded.

Expected behavior
What you expected to happen.

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the custom builder's _split_generators and _generate_examples methods in the issue, then run TFDS preparation with the local LR and HR paths. Verify that the paths are accessible and that the walk finds matching PNG files in both folders. Done means the builder yields examples and preparation completes without the no-examples assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.