aai-institute / aai-institute/pyDVL

joblib's prefetching interferes with skip_indices

Open
#664 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
146
Forks
10
PR merge metrics
No merged PRs in 30d

Description

Under some circumstances joblib's pre-fetching from the batch_generator inside `SemiValueValuation.fit()` consumes way more samples than instructed, even after passing `batch_size=1` and `pre_dispatch="n_jobs"` in the call to `Parallel`. This renders the mechanism that tries to interrupt the processing by interrupting the sampler via `skip_indices` useless.

A good way to test this is to use a powerset sampler with a sequential index iteration which generates, say 1000 samples per index. Then use `MinUpdates(10)` as stopping criterion. Even though skip_indices will be correctly updated to skip the first index after a few updates, the processing will continue for a long while before moving on to the next index.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.