tensorflow / tensorflow/datasets

Unknown error appears when I use the UCF101 dataset. Perhaps some bug exists in the file extractor.py.

Open
#2,539 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Short description
when I try to use the UCF101 dataset ,the program report something like this
tensorflow.python.framework.errors_impl.OutOfRangeError: E:\tfdsdata\datasets\ucf\downloads\thumos14_files_UCF101_videosxm55JXkGdBSDxwckqpN5c7GNr_LXm9dTyoJdpxR_aas.zip; Unknown error

Environment information

  • Operating System: Win10

  • Python version: 3.7(Conda)

  • tensorflow-datasets/tfds-nightly version: tensorflow-datasets 3.2.1

  • tensorflow/tf-nightly version: tensorflow-gpu 2.3.1

  • Does the issue still exists with the last tfds-nightly package (pip install --upgrade tfds-nightly) ?
    yes

Reproduction instructions

mnist_train = tfds.load(name="ucf101", data_dir="E:\\tfdsdata\\datasets\\ucf")

or just reproduce the problem like this:

#something.zip refers to any zipFile
import tensorflow.compat.v2 as tf
with tf.io.gfile.GFile('E:\\tfdsdata\\datasets\\ucf\\downloads\\something.zip', 'r') as f_obj:
    z = zipfile.ZipFile(f_obj)

Link to logs

Expected behavior
I looked into the extractor.py file and fond the reason:
It seems that when zipfile.ZipFile() trys to unzip a file which is wrapped by tf.io.gfile.GFile, it throws an exception.

@contextlib.contextmanager
def _open_or_pass(path_or_fobj):
  if isinstance(path_or_fobj, six.string_types):
    with tf.io.gfile.GFile(path_or_fobj, 'rb') as f_obj:
      yield f_obj
  else:
    yield path_or_fobj

I manage to solve this problem by trying to not use the wrapped file. something like this:

...
def iter_zip(arch_f):
  """Iterate over zip archive."""
  with _open_or_pass(arch_f) as fobj:
    ########
    z = zipfile.ZipFile(fobj)#change this
    ########
    for member in z.infolist():
      extract_file = z.open(member)
      if member.is_dir():  # Filter directories  # pytype: disable=attribute-error
        continue
      path = _normpath(member.filename)
      if not path:
        continue
      yield [path, extract_file]
def iter_zip(arch_f):
  """Iterate over zip archive."""
  with _open_or_pass(arch_f) as fobj:
    ########
    z = zipfile.ZipFile(arch_f)
    ########
    for member in z.infolist():
      extract_file = z.open(member)
      if member.is_dir():  # Filter directories  # pytype: disable=attribute-error
        continue
      path = _normpath(member.filename)
      if not path:
        continue
      yield [path, extract_file]

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the UCF101 load or standalone ZIP example on Windows, then inspect extractor.py and its iter_zip function, especially the interaction between tf.io.gfile.GFile and zipfile.ZipFile. Done means the reported ZIP extraction reproductions complete without the Unknown error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.