huggingface / huggingface/lighteval
[FT] showing count in Markdown summary table
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Issue encountered
I am seeing this output from `lighteval endpoint litellm` with `lighteval==0.10.0`:
```none
| Task |Version| Metric |Value| |Stderr|
|------------------------|------:|---------------|----:|---|-----:|
|all | |ether0_accuracy| 0|± | 0|
|community:ether0:loose:0| 0|ether0_accuracy| 0|± | 0|
```
At a glance, it's unclear if I have misconfigured `LightevalTaskConfig.hf_filter`, or if accuracy truly was 0%.
(Note: in actuality, there were 10 questions, all incorrect.)
## Solution/Feature
Showing a count of questions run, to clarify the size of the evaluation.
## Possible alternatives
None
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.