aai-institute / aai-institute/pyDVL

Implement two-level caching on utility (level 1) and subset of indicies (level 2).

オープン
#475 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
146
フォーク
10
PR マージ指標
30日以内にマージされた PR はありません

説明

Running a benchmark with multiple experiments results in similar calls to the utility object. In between experiments one has to decide if the cache needs to be emptied or not.

Despite the same samples CWS can't use the same cache as TMC. Abstractly speaking it might happen that there are multiple experiments `E1`, `E2`, `E3`. Assume the transition form `E1` to `E2` invalidates the cache, but `E3` can reuse the cache. By imposing the algorithmic knowledge when to invalidate caches on the user we disable the access to certain features to others. It would be favorable if there are multiple experiment caches. These high level caches might use a `LRU(10)` policy (the sub objects might use a `LRU(10000)` policy).

Verifying, that the same cache can be reused from a historic experiment `EH`, can be done with a signature of the utility object. As the value of the utility depends on the model, dataset and scorer. It also depends on the valuation method in our current state (CWS imposes a modified scorer), but this would be caught by estimating the equivalence of the scorer in the signature.

So my proposal would be to add:

- [ ] Calculate signature of utility function to be used for caching
- [ ] (Optional) Signature is a premature check and might be improved by using statistic sampling of input and output values
- [ ] Pass two numbers, `n_low_level` and `n_high_level` for specifying size of both `LRU` policies.
- [ ] Update documentation

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。