aai-institute / aai-institute/pyDVL
joblib's prefetching interferes with skip_indices
- 主要语言
- Python
- 星标
- 146
- 派生
- 10
- PR 合并指标
- 30 天内没有已合并 PR
描述
Under some circumstances joblib's pre-fetching from the batch_generator inside `SemiValueValuation.fit()` consumes way more samples than instructed, even after passing `batch_size=1` and `pre_dispatch="n_jobs"` in the call to `Parallel`. This renders the mechanism that tries to interrupt the processing by interrupting the sampler via `skip_indices` useless.
A good way to test this is to use a powerset sampler with a sequential index iteration which generates, say 1000 samples per index. Then use `MinUpdates(10)` as stopping criterion. Even though skip_indices will be correctly updated to skip the first index after a few updates, the processing will continue for a long while before moving on to the next index.
贡献指南
调研方向
Look at the batch_generator inside SemiValueValuation.fit() and how it interacts with joblib's Parallel prefetching. The test case involves a powerset sampler with sequential index iteration generating many samples per index and MinUpdates(10) as a stopping criterion. Examine the skip_indices mechanism and joblib's pre_dispatch parameter to understand why prefetching consumes extra samples. Running a test with these conditions will show the issue.
由索引模型根据 Issue 内容生成。
评估
- 领域
- data-engineering, machine-learning
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 45/100