FNNDSC / FNNDSC/chris_plugin

Feature request: PathMapper trickle iteration

Open
#6 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
0
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Depends on #5

Often, the items from a `PathMapper` are fed into a `concurrent.futures.ThreadPoolExecutor`. We should think of a way of how we can throttle the iteration, to stay in a safe zone under the memory limit.

Pseudocode:

1. yield first item
2. while not empty...
3. sleep 5 seconds
4. how much did memory usage increase? if available memory is greater than average memory increase per item, then yield next item.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.