Propose to add note about default behaviour of TFRecord Reader to documentation
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 5.8k
- Forks
- 678
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 27
Description
Hi,
I'm trying to use TFRecord Reader instead of FileReader and there is a significant difference in model accuracy. After some debugging and reading the issues it appears that the problem appears because TFRecordReader expects samples to be shuffled. In my case setting initial_fill to be equal to size of the dataset fixed the issue
The problem is, that the fact that DALI expects TFRecords to be preshuffled is only mentioned in several issues on GitHub: https://github.com/NVIDIA/DALI/issues/2849#issuecomment-815996189 and https://github.com/NVIDIA/DALI/issues/1774#issuecomment-592431495 while in the official documentation there is nothing about it. Such behaviour is confusing and may lead to undesired behaviour.
So I'm proposing to add information about pre-shuffle to TFRecordReader class documentation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the TFRecordReader class documentation and review the linked GitHub discussions for the documented behavior. Update the documentation to explain that DALI expects TFRecords to be preshuffled and mention the relevant initial_fill consideration; done means users can find this guidance in the official class documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- tensorflow
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100