huggingface / huggingface/lighteval

Balanced few-shot sampling truncates on a falsy class label and uses the global random module

Open
#1,309 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Bug

`FewShotSampler._init_fewshot_sampling_balanced` (`src/lighteval/tasks/prompt_manager.py`) has two issues in the same function.

### 1. A falsy class label silently truncates the balanced sample

The selection loop guards the label cycle with `if not next_label`:

```python
labels_iterable = cycle(sorted_labels)
while num_instances_to_sample > 0:
next_label = next(labels_iterable, None)
if not next_label:
break
...
```

`labels_iterable` is a `cycle`, which never yields `None` for a non-empty pool, so the intent of the guard is "stop when there are no labels". But `if not next_label` also fires for a label that is present but *falsy*: the integer `0`, an empty string, `False`. Labels come from `instance.fewshot_sorting_class or as_list(instance.get_golds())[0]`, so any task whose class label is a falsy value hits this, e.g. a classification task with an integer `0` label, or a gold that is the empty string. When the cycle reaches that label, the loop breaks and the balanced selection is cut short, down to zero examples if the falsy label sorts first.

### 2. Balanced sampling uses the global `random` module

Unlike the sibling `_init_fewshot_sampling_random` (which uses a local `random.Random(variance_seed)`), the balanced path calls `random.seed(variance_seed)`, `random.shuffle(...)`, and `random.randrange(...)` on the global module. That makes the selection non-reproducible if any other code touches the global RNG between calls, and it leaks state into the rest of the program.

## Reproduction

Using the real `FewShotSampler` (only a duck-typed task is stubbed, exposing `fewshot_docs()` / `fewshot_selection` / `fewshot_split`):

```python
from lighteval.tasks.prompt_manager import FewShotSampler
from lighteval.tasks.requests import Doc
import random

class FakeTask:
fewshot_selection = "balanced"
fewshot_split = "test"
def __init__(self, docs): self._docs = docs
def fewshot_docs(self): return self._docs

# one class has the empty-string gold -> falsy label
docs = [Doc(query=f"q{i}", choices=["", "x"], gold_index=0) for i in range(20)]
docs += [Doc(query=f"p{i}", choices=["", "x"], gold_index=1) for i in range(20)]
selected = FewShotSampler(FakeTask(docs)).sample_fewshot_examples(10, variance_seed=0)
print(len(selected)) # 0 (expected 10)

# global RNG pollution
random.seed(12345); before = [random.random() for _ in range(3)]
random.seed(12345)
FewShotSampler(FakeTask([Doc(query=f"q{i}", choices=["a","b"], gold_index=i%2) for i in range(20)])).sample_fewshot_examples(5, variance_seed=999)
after = [random.random() for _ in range(3)]
print(before == after) # False (the global stream was disturbed)
```

Output on current `main`: `0` balanced examples for the falsy-label case, and `False` for the RNG check.

## Suggested fix

1. Change `if not next_label` to `if next_label is None` (a `cycle` only yields `None` when `sorted_labels` is empty, which is the case the guard was meant for; falsy-but-valid labels no longer truncate).
2. Use a local `rnd = random.Random(variance_seed)` and its `.shuffle` / `.randrange`, mirroring `_init_fewshot_sampling_random`.

Happy to open a PR with both plus regression tests (a falsy-label task gets the full `num_fewshot`, and a balanced call leaves the global RNG stream untouched).

## Environment

Reproduced against `huggingface/lighteval` `main` (current). Pure Python, no model needed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/lighteval/tasks/prompt_manager.py at FewShotSampler._init_fewshot_sampling_balanced, comparing it with _init_fewshot_sampling_random and the supplied falsy-label and global-RNG reproductions. Add regression coverage showing balanced sampling reaches num_fewshot for falsy labels and leaves the global random stream unchanged; the issue is done when both cases pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.