huggingface / huggingface/lighteval

[FT] showing count in Markdown summary table

Open
#804 5 comments 1 reaction 0 assignees View on GitHub
feature good first issue
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Issue encountered

I am seeing this output from `lighteval endpoint litellm` with `lighteval==0.10.0`:

```none
| Task |Version| Metric |Value| |Stderr|
|------------------------|------:|---------------|----:|---|-----:|
|all | |ether0_accuracy| 0|± | 0|
|community:ether0:loose:0| 0|ether0_accuracy| 0|± | 0|
```

At a glance, it's unclear if I have misconfigured `LightevalTaskConfig.hf_filter`, or if accuracy truly was 0%.

(Note: in actuality, there were 10 questions, all incorrect.)

## Solution/Feature

Showing a count of questions run, to clarify the size of the evaluation.

## Possible alternatives

None

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.