NVIDIA / NVIDIA/DALI

How can I make DALI load tf.SequenceExample records?

Open
#3,824 4 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

enhancement external contribution welcome external source
Dominant language
C++
Stars
5.8k
Forks
678
Avg merge
3d 1h
Merged PRs (30d)
27

Description

Hi,

Running into an issue with DALI. I'm working with a dataset stored in the format of https://www.tensorflow.org/api_docs/python/tf/train/SequenceExample. Other tfrecord readers handle it fine, such as https://github.com/vahidk/tfrecord.

The error is on reading the index file Assert on "p != nullptr" failed: Error reading from a file {FILE} the file is valid however the index produced is somewhat odd in that it is just a single line of 0 152207822 despite this being many records. The indexer in the above tool produces the same result but can still load it fine.

The failing part:

inputs = fn.readers.tfrecord(
   path=all_files,
   index_path=indices,
   features={
    # Doesn't actually matter which feature I try to pull out.
     "image/encoded" : tfrec.VarLenFeature(tfrec.string, "")},
    },  num_shards=1, name='Reader')
dali/dali/operators/reader/loader/indexed_file_loader.h:76

DALI version comes from https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch using tag 22.03-py3

Any advice? Are these tf.SequenceExample just not supported in DALI? Are they on a roadmap?

FYI: The PyTorch loader above succeeds but takes in a separate argument for sequential features.

tfrecord.tfrecord_loader(path, None, sequence_description={"image/encoded": "byte"})

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.