tensorflow / tensorflow/datasets

Validate TFRecords after data generation for corrupted/truncated records

Open
#3,330 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Is your feature request related to a problem? Please describe.
Sometimes when generating a dataset using tfds.load the data stored as TFRecords will get truncated or image data will get corrupted in the process but the dataset generation will continue without raising any flags.

It is not until a user begins training a model on the dataset that Tensorflow raises the DataLossError and notifies the user that the data has been either truncated or corrupted.

Especially for large datasets, it's annoying to spend a long time generating data, building a model, and training data only to be stopped by this error at the first step.

There is a way for the user to validate the TFRecords individually by creating tf.data.TFRecordDataset objects for each record, but a user would not think to do that unless they had encountered this problem in training.

Describe the solution you'd like
I would like a way for the user to know that their TFRecords have been corrupted or truncated at the data generation step.

Describe alternatives you've considered
A simple way to validate each TFRecord:

filenames = os.listdir('path/to/records/1.0.0')
i = 0
for fname in filenames:
  print('validating ', fname)
  records = tf.data.TFRecordDataset('path/to/records/1.0.0'+fname)
  try:
      for byte in records:
          i += 1
  except Exception as e:
      print('error in {} at record {}'.format(fname, i))
      print(e)

I would like a similar method implemented in tfds.load after the records have been made that at least gives users a warning about the corrupted/truncated data.

Additional context
There is a way to bypass these errors during training by implementing tf.data.experimental.ignore_errors but it can be problematic because it ignores other errors that could be important and should only be used as a workaround.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the dataset-generation completion path behind tfds.load and compare it with the issue's TFRecordDataset validation example. Determine where validation could report truncated or corrupted records without masking unrelated errors; done means generation warns users when record validation fails. The issue names no repository files or tests, so locating suitable entry points and coverage is part of the work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.