aai-institute / aai-institute/pyDVL

Simplify caching

Open
#600 2 comments 0 reactions 0 assignees View on GitHub
cleanup design-problem
Dominant language
Python
Stars
146
Forks
10
PR merge metrics
No merged PRs in 30d

Description

The new design of data valuation methods avoids repeated computations of the utility function without relying on caching. We could therefore get rid of our current caching implementation based on memcached, which seems overpowered. This would close several issues related to caching (e.g. #517, #475, #464 and #459). Moreover, it could solve problems that arise due to the many files the current caching solution creates.

The only situation where caching ist still really important is when one benchmarks multiple algorithms and wants to use caching to ensure that randomness is kept as constant as possible between different algorithms and to save runtime in the benchmark. We therefore should create an entry point for benchmarking frameworks to enable caching. I see two possible solutions:

1. Use a simple shared-memory cache to store all utility evaluations and return them as part of the ValuationResult. A benchmarking library could then use these evaluations to build up a cache. All logic to wrap Utility with a cached version would be in the benchmarking library.
2. We could keep the `cache_backend` abstraction in the Utility but only implement a much simpler shared-memory backend in pydvl. Users with advanced caching needs could then build their own backends.

Contributor guide

Open the contributing guide

Research direction

Look at the current caching implementation using memcached, likely in the utility module. Understand the `cache_backend` abstraction and the ValuationResult. The goal is to remove the complex caching, possibly replacing it with a simple shared-memory cache for benchmarking. Check linked issues #517, #475, #464, #459 for context. 'Done' means caching is simplified, the memcached dependency is removed, and a new entry point for benchmarking frameworks exists.

Written by the indexing model from the issue text.

Assessment

Tech stack
memcached
Domain
backend, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.