EpistasisLab / EpistasisLab/tpot
Cache cleanup during long runs
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
I can easily run out of disk space during a long TPOT run with caching enabled. Caching is hugely useful, but needs some periodic management.
## Context of the issue
The joblib.Memory class supports size limits and LRU clearing, but that requires an explicit call to the memory object's `reduce_size()` method. As best I can tell from grepping the codebases, neither tpot nor sklearn calls `reduce_size()`. During a 12 hour run yesterday, the cache grew to 1.3TB and filled my disk. It would be nice to have some mechanism to periodically call `reduce_size()` if the memory object has `bytes_limit` set.
In the meanwhile, the best workaround I can think of is to write a tiny python script with an identical Memory that calls `reduce_size()`. I'd run that as a separate python process every hour or so. Or if I could automate it a little, by having my main script fork the housekeeping as a background process that gets terminated via `atexit` when the main script is finished.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.