Cache open HDF5 file handles in DataLoader workers
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 1
Description
The `dataset_type=1` reader opens the shard file, reads one sample, and closes the file on every `__getitem__`. HDF5 files can hold many samples with this dataset type, so the same file gets reopened many times per epoch
**Proposed change**
Each DataLoader worker keeps a small pool of recently used open HDF5 files (LRU cache via opt-in via CLI flag?).
Reads go through the pooled handle when the file is already open. Least recently used handle is closed when the pool is full.
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the dataset_type=1 reader and the DataLoader worker setup, then trace how shard files are opened and closed during __getitem__. Determine the intended CLI opt-in behavior and worker-local cache limits. Done means repeated reads can reuse open handles, the least recently used handle is closed when the pool is full, and the relevant behavior is tested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100