aai-institute / aai-institute/pyDVL
joblib's prefetching interferes with skip_indices
- Dominant language
- Python
- Stars
- 146
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
Under some circumstances joblib's pre-fetching from the batch_generator inside `SemiValueValuation.fit()` consumes way more samples than instructed, even after passing `batch_size=1` and `pre_dispatch="n_jobs"` in the call to `Parallel`. This renders the mechanism that tries to interrupt the processing by interrupting the sampler via `skip_indices` useless.
A good way to test this is to use a powerset sampler with a sequential index iteration which generates, say 1000 samples per index. Then use `MinUpdates(10)` as stopping criterion. Even though skip_indices will be correctly updated to skip the first index after a few updates, the processing will continue for a long while before moving on to the next index.
Contributor guide
Assessment
This issue has not been assessed yet.