EpistasisLab / EpistasisLab/tpot

Cache cleanup during long runs

Open
#998 4 comments 0 reactions 0 assignees View on GitHub
enhancement need contributor
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

I can easily run out of disk space during a long TPOT run with caching enabled. Caching is hugely useful, but needs some periodic management.

## Context of the issue

The joblib.Memory class supports size limits and LRU clearing, but that requires an explicit call to the memory object's `reduce_size()` method. As best I can tell from grepping the codebases, neither tpot nor sklearn calls `reduce_size()`. During a 12 hour run yesterday, the cache grew to 1.3TB and filled my disk. It would be nice to have some mechanism to periodically call `reduce_size()` if the memory object has `bytes_limit` set.

In the meanwhile, the best workaround I can think of is to write a tiny python script with an identical Memory that calls `reduce_size()`. I'd run that as a separate python process every hour or so. Or if I could automate it a little, by having my main script fork the housekeeping as a background process that gets terminated via `atexit` when the main script is finished.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.