llnl / llnl/professor

Cache open HDF5 file handles in DataLoader workers

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
2
Forks
1
Avg merge
4d 21h
Merged PRs (30d)
1

Description

The `dataset_type=1` reader opens the shard file, reads one sample, and closes the file on every `__getitem__`. HDF5 files can hold many samples with this dataset type, so the same file gets reopened many times per epoch

**Proposed change**

Each DataLoader worker keeps a small pool of recently used open HDF5 files (LRU cache via opt-in via CLI flag?).
Reads go through the pooled handle when the file is already open. Least recently used handle is closed when the pool is full.

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the dataset_type=1 reader and the DataLoader worker setup, then trace how shard files are opened and closed during __getitem__. Determine the intended CLI opt-in behavior and worker-local cache limits. Done means repeated reads can reuse open handles, the least recently used handle is closed when the pool is full, and the relevant behavior is tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.