aai-institute / aai-institute/pyDVL

joblib's prefetching interferes with skip_indices

未关闭
#664 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
146
派生
10
PR 合并指标
30 天内没有已合并 PR

描述

Under some circumstances joblib's pre-fetching from the batch_generator inside `SemiValueValuation.fit()` consumes way more samples than instructed, even after passing `batch_size=1` and `pre_dispatch="n_jobs"` in the call to `Parallel`. This renders the mechanism that tries to interrupt the processing by interrupting the sampler via `skip_indices` useless.

A good way to test this is to use a powerset sampler with a sequential index iteration which generates, say 1000 samples per index. Then use `MinUpdates(10)` as stopping criterion. Even though skip_indices will be correctly updated to skip the first index after a few updates, the processing will continue for a long while before moving on to the next index.

贡献指南

打开贡献指南

调研方向

Look at the batch_generator inside SemiValueValuation.fit() and how it interacts with joblib's Parallel prefetching. The test case involves a powerset sampler with sequential index iteration generating many samples per index and MinUpdates(10) as a stopping criterion. Examine the skip_indices mechanism and joblib's pre_dispatch parameter to understand why prefetching consumes extra samples. Running a test with these conditions will show the issue.

由索引模型根据 Issue 内容生成。

评估

领域
data-engineering, machine-learning
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。