tensorflow / tensorflow/datasets
Validate TFRecords after data generation for corrupted/truncated records
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Is your feature request related to a problem? Please describe.
Sometimes when generating a dataset using tfds.load the data stored as TFRecords will get truncated or image data will get corrupted in the process but the dataset generation will continue without raising any flags.
It is not until a user begins training a model on the dataset that Tensorflow raises the DataLossError and notifies the user that the data has been either truncated or corrupted.
Especially for large datasets, it's annoying to spend a long time generating data, building a model, and training data only to be stopped by this error at the first step.
There is a way for the user to validate the TFRecords individually by creating tf.data.TFRecordDataset objects for each record, but a user would not think to do that unless they had encountered this problem in training.
Describe the solution you'd like
I would like a way for the user to know that their TFRecords have been corrupted or truncated at the data generation step.
Describe alternatives you've considered
A simple way to validate each TFRecord:
filenames = os.listdir('path/to/records/1.0.0')
i = 0
for fname in filenames:
print('validating ', fname)
records = tf.data.TFRecordDataset('path/to/records/1.0.0'+fname)
try:
for byte in records:
i += 1
except Exception as e:
print('error in {} at record {}'.format(fname, i))
print(e)
I would like a similar method implemented in tfds.load after the records have been made that at least gives users a warning about the corrupted/truncated data.
Additional context
There is a way to bypass these errors during training by implementing tf.data.experimental.ignore_errors but it can be problematic because it ignores other errors that could be important and should only be used as a workaround.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the dataset-generation completion path behind tfds.load and compare it with the issue's TFRecordDataset validation example. Determine where validation could report truncated or corrupted records without masking unrelated errors; done means generation warns users when record validation fails. The issue names no repository files or tests, so locating suitable entry points and coverage is part of the work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100