lincc-frameworks / lincc-frameworks/hyrax
OOM safety for dataloaders
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 41
- Forks
- 7
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 8
Description
If a Train/Infer run is estimated to have a size of data that exceeds the available system or GPU memory, warn the user and suggest alternatives to reduce the data size.
This could be several fold:
1) Near the memory limit we warn the user that it *may* exceed memory limits, but allow the run with no changes to settings.
2) at or above the memory limit for CPU memory, we disable data_loader caching or cap the size of the cache at near system memory to avoid an OOM kill.
3) If a batch is exceeding GPU memory, we warn the user and suggest an appropriate batch size for their data.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Train/Infer dataloader paths and any existing data-size estimates. Review how data_loader caching, CPU memory, GPU memory, and batch size are handled. Done means runs warn about near-limit memory, avoid unsafe CPU caching, and suggest a smaller batch size when GPU memory is exceeded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100