huggingface / huggingface/lighteval
How to print all pass@k scores when generating 16 samples?
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Hi,
I want to print all results of pass@k metrics when generating 16 samples. (e.g., k=1, 2, 4, 8, 16)
```python
math_500_pass_k_at_16 = LightevalTaskConfig(
name="math_500_pass_k_at_16",
suite=["custom"],
prompt_function=math_500_prompt_fn,
hf_repo="HuggingFaceH4/MATH-500",
hf_subset="default",
hf_avail_splits=["test"],
evaluation_splits=["test"],
few_shots_split=None,
few_shots_select=None,
generation_size=32768,
metrics=[
Metrics.pass_at_k_math(sample_params={"k": 1, "n": 16}),
Metrics.pass_at_k_math(sample_params={"k": 2, "n": 16}),
Metrics.pass_at_k_math(sample_params={"k": 4, "n": 16}),
Metrics.pass_at_k_math(sample_params={"k": 8, "n": 16}),
Metrics.pass_at_k_math(sample_params={"k": 16, "n": 16}),
],
version=2,
```
But, I can't see full results that I want. Does anyone know how to resolve it?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.