How can I make DALI load tf.SequenceExample records?
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 5.8k
- Forks
- 678
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 27
Description
Hi,
Running into an issue with DALI. I'm working with a dataset stored in the format of https://www.tensorflow.org/api_docs/python/tf/train/SequenceExample. Other tfrecord readers handle it fine, such as https://github.com/vahidk/tfrecord.
The error is on reading the index file Assert on "p != nullptr" failed: Error reading from a file {FILE} the file is valid however the index produced is somewhat odd in that it is just a single line of 0 152207822 despite this being many records. The indexer in the above tool produces the same result but can still load it fine.
The failing part:
inputs = fn.readers.tfrecord(
path=all_files,
index_path=indices,
features={
# Doesn't actually matter which feature I try to pull out.
"image/encoded" : tfrec.VarLenFeature(tfrec.string, "")},
}, num_shards=1, name='Reader')
dali/dali/operators/reader/loader/indexed_file_loader.h:76
DALI version comes from https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch using tag 22.03-py3
Any advice? Are these tf.SequenceExample just not supported in DALI? Are they on a roadmap?
FYI: The PyTorch loader above succeeds but takes in a separate argument for sequential features.
tfrecord.tfrecord_loader(path, None, sequence_description={"image/encoded": "byte"})
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.