aai-institute / aai-institute/pyDVL

Implement two-level caching on utility (level 1) and subset of indicies (level 2).

未关闭
#475 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
146
派生
10
PR 合并指标
30 天内没有已合并 PR

描述

Running a benchmark with multiple experiments results in similar calls to the utility object. In between experiments one has to decide if the cache needs to be emptied or not.

Despite the same samples CWS can't use the same cache as TMC. Abstractly speaking it might happen that there are multiple experiments `E1`, `E2`, `E3`. Assume the transition form `E1` to `E2` invalidates the cache, but `E3` can reuse the cache. By imposing the algorithmic knowledge when to invalidate caches on the user we disable the access to certain features to others. It would be favorable if there are multiple experiment caches. These high level caches might use a `LRU(10)` policy (the sub objects might use a `LRU(10000)` policy).

Verifying, that the same cache can be reused from a historic experiment `EH`, can be done with a signature of the utility object. As the value of the utility depends on the model, dataset and scorer. It also depends on the valuation method in our current state (CWS imposes a modified scorer), but this would be caught by estimating the equivalence of the scorer in the signature.

So my proposal would be to add:

- [ ] Calculate signature of utility function to be used for caching
- [ ] (Optional) Signature is a premature check and might be improved by using statistic sampling of input and output values
- [ ] Pass two numbers, `n_low_level` and `n_high_level` for specifying size of both `LRU` policies.
- [ ] Update documentation

贡献指南

打开贡献指南

调研方向

The issue describes adding two-level LRU caching to utility objects in pyDVL. Start by locating the utility object implementation and its current caching mechanism. Investigate how experiments are run and where utility calls are made. The signature for cache reuse likely involves hashing the model, dataset, and scorer. Look for existing LRU cache implementations or consider using functools.lru_cache. Update documentation to explain the new caching parameters and behavior.

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
machine-learning, performance
Issue 类型
功能
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。