NVIDIA / NVIDIA/DALI

Propose to add note about default behaviour of TFRecord Reader to documentation

Open
#3,088 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
C++
Stars
5.8k
Forks
678
Avg merge
3d 1h
Merged PRs (30d)
27

Description

Hi,

I'm trying to use TFRecord Reader instead of FileReader and there is a significant difference in model accuracy. After some debugging and reading the issues it appears that the problem appears because TFRecordReader expects samples to be shuffled. In my case setting initial_fill to be equal to size of the dataset fixed the issue

The problem is, that the fact that DALI expects TFRecords to be preshuffled is only mentioned in several issues on GitHub: https://github.com/NVIDIA/DALI/issues/2849#issuecomment-815996189 and https://github.com/NVIDIA/DALI/issues/1774#issuecomment-592431495 while in the official documentation there is nothing about it. Such behaviour is confusing and may lead to undesired behaviour.

So I'm proposing to add information about pre-shuffle to TFRecordReader class documentation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the TFRecordReader class documentation and review the linked GitHub discussions for the documented behavior. Update the documentation to explain that DALI expects TFRecords to be preshuffled and mention the relevant initial_fill consideration; done means users can find this guidance in the official class documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
tensorflow
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.