EpistasisLab / EpistasisLab/tpot

New feature: MongoDB or other NOSQL for memory

Open
#928 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

The "memory" in TPOT caches candidate pipeline metadata on disk.
The data is cached as a large number of very small ASCII text files each containing data represented by a JSON object.

It occurs to me that this is a crude "document database", and both time and space efficiencies could be gained by using something like MongoDB instead.

I have also discovered my "Real Time Protection" anti-malware software is scanning every file in the cache. This has a devastating impact on performance!

Recommend that users with any anti-virus software installed should configure anti-malware settings to white-list (exclude) the location of the joblib memory cache for performance.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.