huggingface / huggingface/lighteval
Sequential few-shot selection mutates the task's shared few-shot pool in place, corrupting later variance seeds
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
### Summary
`FewShotSampler._init_fewshot_sampling_sequential` rotates the list returned by `task.fewshot_docs()` in place. That list is the task's memoized `_fewshot_docs`, returned by reference, so the rotation mutates shared state. When an evaluation runs several few-shot seeds against the same task (variance estimation, i.e. `few_shot_iterations > 1`), each seed rotates the already-rotated pool, so the offsets accumulate and every seed after the first selects the wrong examples. Already-built cache entries are corrupted too, because `self._fewshot_cache[variance_seed]` stores a reference to the same shared list instead of a copy, so all seeds end up aliasing one over-rotated list.
The sibling `_init_fewshot_sampling_random` already copies the pool with `list(...)` before touching it; the sequential path does not.
Only the `sequential` selection method is affected. `random` copies the pool, and `balanced` builds its own per-label lists and never mutates the pool.
### Reproduction
```python
from lighteval.tasks.lighteval_task import LightevalTask, LightevalTaskConfig
from lighteval.tasks.prompt_manager import FewShotSampler
from lighteval.tasks.requests import Doc
config = LightevalTaskConfig(
name="demo", prompt_function=lambda _, __: None, hf_repo="", hf_subset="default",
metrics=[], few_shots_split="test", few_shots_select="sequential",
)
task = LightevalTask(config)
task._fewshot_docs = [Doc(query=str(i), choices=["A", "B"], gold_index=0) for i in range(10)]
original = [d.query for d in task._fewshot_docs]
sampler = FewShotSampler(task)
# Variance evaluation scores the same item with several few-shot seeds.
for seed in (0, 1, 2):
picked = [d.query for d in sampler.sample_fewshot_examples(num_fewshot=2, variance_seed=seed)]
rot = (2 * seed) % len(original)
expected = (original[rot:] + original[:rot])[:2]
print(f"seed {seed}: picked={picked} expected={expected}")
print("pool after sampling:", [d.query for d in task.fewshot_docs()])
print("pool unchanged:", [d.query for d in task.fewshot_docs()] == original)
```
Output on current `main`:
```
seed 0: picked=['0', '1'] expected=['0', '1']
seed 1: picked=['2', '3'] expected=['2', '3']
seed 2: picked=['6', '7'] expected=['4', '5']
pool after sampling: ['6', '7', '8', '9', '0', '1', '2', '3', '4', '5']
pool unchanged: False
```
Seed 2 should select the pool rotated by 4 (`['4', '5']`) but gets a rotation of 6 (`['6', '7']`), because seed 1 already rotated the shared pool by 2 and seed 2 rotates that result by 4 more. The pool itself is left rotated after the run.
### Root cause
In `src/lighteval/tasks/prompt_manager.py`:
```python
def _init_fewshot_sampling_sequential(self, num_fewshot, variance_seed):
fewshotpool = self.task.fewshot_docs() # memoized list, returned by reference
for _ in range(num_fewshot * variance_seed):
fewshotpool.append(fewshotpool.pop(0)) # rotates the shared pool in place
self._fewshot_cache[variance_seed] = fewshotpool
```
`fewshot_docs()` returns `self._fewshot_docs`, so the in-place rotation persists across calls and every cache entry aliases the same list.
### Expected behavior
Sequential selection for seed `s` should return the pool rotated by `num_fewshot * s` from the original order, independently per seed, and leave `task.fewshot_docs()` unchanged.
### Fix
Copy the pool before rotating, matching `_init_fewshot_sampling_random`:
```python
fewshotpool = list(self.task.fewshot_docs())
```
I have a patch plus a regression test ready and will open a PR.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/lighteval/tasks/prompt_manager.py by reading _init_fewshot_sampling_sequential and comparing it with _init_fewshot_sampling_random. Run the reproduction from the issue and review the available regression test; done means each variance seed uses the original pool independently and task.fewshot_docs() remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100