huggingface / huggingface/lighteval
Sample cache: doc.id collides across evaluation splits, crashing runs and silently serving one split's answers for another split's questions
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
PUBLIC-READY
ETB-15 (artifact-blind rebind) with the R01E1-M1 positional-rebinding mechanism: a cache joined on a positional key that is not unique across the two index spaces that produce it
## What happens
`LightevalTask._get_docs_from_split` assigns `doc.id = str(ix)` where `ix` comes from an `enumerate()` that restarts for every split in `evaluation_splits` (src/lighteval/tasks/lighteval_task.py, `_get_docs_from_split`, around line 314). So when a task evaluates more than one split, the same id exists once per split and refers to different questions.
`SampleCache` stores and serves cached model responses under the key (task full name, task hash, sampling method, `doc.id`) (src/lighteval/utils/cache_management.py, `get_samples_to_process_and_cache`, `cache_samples`, `get_samples_from_cache`). Nothing in the key or in the stored row identifies the split or the content of the question.
Two consequences, both reproduced at pin 932e1f2f4c5af3e926534f12b2a84a3ae18d6d3f:
1. Crash plus persistent cache corruption. When both owners of a colliding id are processed in the same call, `cache_samples` writes two rows with the same `sample_id`, and `get_samples_from_cache` then does `dataset_df.loc[doc.id]`, which returns a two-row frame instead of a row. `_load_sample` feeds that to `ModelResponse(**...)` and the run dies with `TypeError: ModelResponse() got multiple values for keyword argument '2'`. The crash happens after the model has already produced its outputs, and the duplicated rows stay in the parquet, so every later run on the same cache keeps crashing until the cache directory is deleted by hand.
2. Silent cross-split rebind. When the set of selected ids changes between two runs (in the PoC: the dataset content changed upstream, which changes the seeded shuffle and the selection; a different `--max_samples` has the same effect), an id that was cached while owned by split A is served for the question that currently sits at that position in split B. The PoC shows a validation question being served the answer the model produced for a test question, with one model call for four documents and no warning anywhere.
Tasks shipped in this repository use multi-split evaluation, for example `lsat_qa` (5 task variants with `evaluation_splits=["validation", "test"]`), `lextreme`, and `legal_summarization`, so this is reachable with built-in tasks plus caching enabled.
## Repro
PoC: `poc_cache_split_collision.py` (deterministic, local datasets, no network, no model inference; uses the real `@cached` decorator, the real `SampleCache`, and real parquet read/write). Driver: `REPRO-E1-F1.sh` runs it twice and diffs stdout.
```
PYTHONPATH=: HF_HUB_OFFLINE=1 /usr/bin/python3 poc_cache_split_collision.py
```
Key output:
```
== A.RUN2: CRASHED after model execution with TypeError: ... got multiple values for keyword argument '2'
== PART A persisted cache file ... sample_ids=['3', '0', '1', '4', '2', '2', '5'] (duplicates present: True)
== B.RUN2-dataset-v2: {"doc_id": "1", "split": "validation", "question": "VAL-Q1", "served_text": "ANSWER_TO::TEST-Q1", "matches_own_question": false}
== PART B verdict: VULNERABLE (1 silent mismatches of 4)
```
## Impact
For multi-split tasks with the prediction cache enabled: wasted full model runs that crash at the end and a cache directory that must be deleted manually; and, when the selected subset changes between runs, scores computed from responses produced for different questions than the ones reported, with zero signal in the output. The stored detail rows then also misattribute answers, since the doc and the served response no longer correspond.
## Suggested fix
Make the cache key content-unique per document. Either include the split name in `doc.id` (for example `f"{split}/{ix}"`), or key the cache on a hash of the doc's query and choices rather than the positional id. Additionally, `cache_samples` should refuse to write two rows with the same `sample_id` in one batch (that is where the persistent corruption is created), and `get_samples_from_cache` should fail with a clear message instead of passing a DataFrame slice into `ModelResponse(**...)`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with LightevalTask._get_docs_from_split in src/lighteval/tasks/lighteval_task.py and the cache methods get_samples_to_process_and_cache, cache_samples, and get_samples_from_cache in src/lighteval/utils/cache_management.py. Run poc_cache_split_collision.py through REPRO-E1-F1.sh to observe duplicate rows and cross-split responses. Done means multi-split cache entries remain distinct, duplicate writes are rejected, and cache lookup failures produce a clear error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100