huggingface / huggingface/lighteval
Prediction cache identity is the model config, not the model that ran: weights moving under the same revision name silently reuse the old responses
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
PUBLIC-READY
ETB-15 (name-pinned identity: the cache verifies a mutable name, not the artifact behind it)
## What happens
The cache directory is `//` (src/lighteval/utils/cache_management.py, `SampleCache.__init__` and `get_model_hash`). For hub models the config carries `revision: str = "main"` by default, which is a mutable branch name, and for local models the config carries a path. In both cases the key pins the NAME, not the bytes.
The code already resolves the true commit for the results log: `TransformersModel.__init__` sets `self.model_sha = config.get_model_sha()` (src/lighteval/models/transformers/transformers_model.py, around line 216), and the same resolution exists on the endpoint side. That resolved sha is simply never fed to the cache.
The documentation promises the opposite. docs/source/caching.mdx says: "A new cache is automatically created when: ... Model weights change (different revision, checkpoint, etc.)". With the default `revision="main"`, weights changing under `main` leaves the config, the hash, and the cache directory identical, and run 2 is served run 1's responses while the run's own logging attributes them to the new weights.
Reproduced at pin 932e1f2f4c5af3e926534f12b2a84a3ae18d6d3f. The PoC simulates the weight change deterministically (the stand-in model reads its behavior from a weights file; the two runs use byte-different weights files with an otherwise identical model config, which is exactly the state a moved branch or an overwritten checkpoint directory produces). The real hub is not contacted.
## Repro
PoC: `poc_cache_model_revision_name.py` (two separate processes sharing one cache dir; real `@cached` decorator and real `SampleCache`; no network, no inference). Driver: `REPRO-E1-F3.sh` runs it twice and diffs stdout.
```
PYTHONPATH=: HF_HUB_OFFLINE=1 /usr/bin/python3 poc_cache_model_revision_name.py
```
Key output:
```
== RUN2-weights-v2: weights_sha_prefix=WEIGHTS_b9599b06 model.greedy_until call count = 0
== RUN2-weights-v2: {"question": "Q0", "served_text": "WEIGHTS_b5c0e3eb::ANSWER_TO::Q0", "from_current_weights": false}
== verdict: VULNERABLE (3 stale of 3)
```
## Impact
Continuous-evaluation setups that rerun a model alias on a schedule (nightly runs of `main`, a fine-tuning job overwriting a checkpoint directory, an endpoint redeployed behind the same name) get the previous weights' cached responses with zero calls, while the results log the new model sha next to them. Scores then describe a mixture of two model versions with nothing in the output saying so.
## Suggested fix
Feed the resolved identity into the cache key. `get_model_sha()` already exists: include the resolved commit sha (and for local paths, a digest of the weights files or their mtimes) in the string that `get_model_hash` hashes, or fail loudly when the resolved sha differs from the one recorded in an existing cache directory. At minimum, update caching.mdx so it stops promising weight-change invalidation, and log cache hit counts per run.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with src/lighteval/utils/cache_management.py, especially SampleCache.__init__ and get_model_hash, then trace the resolved identity from TransformersModel.__init__ in src/lighteval/models/transformers/transformers_model.py. Run poc_cache_model_revision_name.py with REPRO-E1-F3.sh and inspect docs/source/caching.mdx. Done means cache reuse is distinguished for changed model artifacts and the documentation matches the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100