Pipeline to iterate over a single NumPy file
Open
@klecki is already working on this.
Since Jan 16, 2025.
question
- Dominant language
- C++
- Stars
- 5.8k
- Forks
- 678
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 27
Description
Describe the question.
I'm trying to use GPU direct storage (GDS) via DALI's numpy reader for a dataset of many (10^4) 3D volumes (each volume is one training sample). However, the API seems to require that one file only contains one sample, so each sample will have to be in a different file, leading to tens of thousands of files. Opening this many files each training epoch could have significant overhead for certain file systems. Is there a way to use larger files instead (for example stacking volumes into chunks) and iterate over a dimension? #4140 suggests using an external source for this, but that would not support GDS.
Check for duplicates
- I have searched the open bugs/issues and have found no duplicates for this bug report
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.