huggingface / huggingface/lighteval

Prediction cache identity is the model config, not the model that ran: weights moving under the same revision name silently reuse the old responses

Open
#1,364 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

PUBLIC-READY
ETB-15 (name-pinned identity: the cache verifies a mutable name, not the artifact behind it)

## What happens

The cache directory is `//` (src/lighteval/utils/cache_management.py, `SampleCache.__init__` and `get_model_hash`). For hub models the config carries `revision: str = "main"` by default, which is a mutable branch name, and for local models the config carries a path. In both cases the key pins the NAME, not the bytes.

The code already resolves the true commit for the results log: `TransformersModel.__init__` sets `self.model_sha = config.get_model_sha()` (src/lighteval/models/transformers/transformers_model.py, around line 216), and the same resolution exists on the endpoint side. That resolved sha is simply never fed to the cache.

The documentation promises the opposite. docs/source/caching.mdx says: "A new cache is automatically created when: ... Model weights change (different revision, checkpoint, etc.)". With the default `revision="main"`, weights changing under `main` leaves the config, the hash, and the cache directory identical, and run 2 is served run 1's responses while the run's own logging attributes them to the new weights.

Reproduced at pin 932e1f2f4c5af3e926534f12b2a84a3ae18d6d3f. The PoC simulates the weight change deterministically (the stand-in model reads its behavior from a weights file; the two runs use byte-different weights files with an otherwise identical model config, which is exactly the state a moved branch or an overwritten checkpoint directory produces). The real hub is not contacted.

## Repro

PoC: `poc_cache_model_revision_name.py` (two separate processes sharing one cache dir; real `@cached` decorator and real `SampleCache`; no network, no inference). Driver: `REPRO-E1-F3.sh` runs it twice and diffs stdout.

```
PYTHONPATH=: HF_HUB_OFFLINE=1 /usr/bin/python3 poc_cache_model_revision_name.py
```

Key output:

```
== RUN2-weights-v2: weights_sha_prefix=WEIGHTS_b9599b06 model.greedy_until call count = 0
== RUN2-weights-v2: {"question": "Q0", "served_text": "WEIGHTS_b5c0e3eb::ANSWER_TO::Q0", "from_current_weights": false}
== verdict: VULNERABLE (3 stale of 3)
```

## Impact

Continuous-evaluation setups that rerun a model alias on a schedule (nightly runs of `main`, a fine-tuning job overwriting a checkpoint directory, an endpoint redeployed behind the same name) get the previous weights' cached responses with zero calls, while the results log the new model sha next to them. Scores then describe a mixture of two model versions with nothing in the output saying so.

## Suggested fix

Feed the resolved identity into the cache key. `get_model_sha()` already exists: include the resolved commit sha (and for local paths, a digest of the weights files or their mtimes) in the string that `get_model_hash` hashes, or fail loudly when the resolved sha differs from the one recorded in an existing cache directory. At minimum, update caching.mdx so it stops promising weight-change invalidation, and log cache hit counts per run.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/lighteval/utils/cache_management.py, especially SampleCache.__init__ and get_model_hash, then trace the resolved identity from TransformersModel.__init__ in src/lighteval/models/transformers/transformers_model.py. Run poc_cache_model_revision_name.py with REPRO-E1-F3.sh and inspect docs/source/caching.mdx. Done means cache reuse is distinguished for changed model artifacts and the documentation matches the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.