huggingface / huggingface/lighteval
[BUG] Very slow livecodebench scoring
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
Much slower scoring compared to livecodebench repo.
## To Reproduce
Compare to LCB repo, scoring takes significantly longer and seems single threaded
## Expected behavior
Match LCB scoring time with same number of threads set. I have tried setting num_process:
```python
def codegen_metric(predictions: list[str], formatted_doc: Doc, **kwargs) -> float:
"""Estimates the Pass@1 metric for the code generation task.
Extract the code from each prediction, Runs it for each sample and generations,
and computes the Pass@1 over the outputs.
"""
# Extract generated code snippets
generated_code_snippets = [[extract_code(pred) for pred in predictions]] # noqa: F841
evaluation_sample = { # noqa: F841
"inputs": formatted_doc.specific["inputs"],
"outputs": formatted_doc.specific["outputs"],
"fn_name": formatted_doc.specific["fn_name"],
}
# This is a list of lists because
evaluation_sample = [{"input_output": json.dumps(evaluation_sample)}]
metrics, _ = codegen_metrics(
evaluation_sample,
generated_code_snippets,
k_list=[1], # Only run for Pass@1
num_process_evaluate=64,
)
return metrics["pass@1"]
```
with no improvement in scoring time
## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.
latest master
```
conda create -n eval python=3.11
pip install vllm==0.7.2
pip install git+https://github.com/huggingface/lighteval.git#egg=lighteval[extended_tasks] math-verify==0.5.2
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.