huggingface / huggingface/lighteval
Lighteval's LCB grader fails correct solutions that read sys.stdin.buffer
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
Lighteval performs the swap `@patch("sys.stdin", StringIO(inputs))` here [link](https://github.com/huggingface/lighteval/blob/78dbee223ba56ce8b7a7a1ac4bbe841f7e10708f/src/lighteval/tasks/tasks/lcb/codegen_metrics.py#L139), which causes code that relies on sys.stdin.buffer.read() to fail.
Livecodebench avoids this issue by swapping in a mock object with a buffer attributes here [link](https://github.com/LiveCodeBench/LiveCodeBench/blob/28fef95ea8c9f7a547c8329f2cd3d32b92c1fa24/lcb_runner/evaluation/testing_util.py#L162-L168).
## To Reproduce
This code reproduces the error and suggests a fix to the call_method.
```
from unittest.mock import mock_open, patch
from io import StringIO
import io
from typing import Callable
from lighteval.tasks.tasks.lcb.codegen_metrics import call_method
def call_method_fixed(method: Callable, inputs: list | str):
if isinstance(inputs, list):
inputs = "\n".join(inputs)
inputs_line_iterator = iter(inputs.split("\n"))
mock_stdin = StringIO(inputs) # <-- new stuff here
mock_stdin.buffer = io.BytesIO(inputs.encode()) # <-- new stuff here
@patch("builtins.open", mock_open(read_data=inputs))
@patch("sys.stdin", mock_stdin) # <-- the fix
@patch("sys.stdin.readline", lambda *args: next(inputs_line_iterator))
@patch("sys.stdin.readlines", lambda *args: inputs.split("\n"))
@patch("sys.stdin.read", lambda *args: inputs)
def _inner_call_method(_method):
try:
return _method()
except SystemExit:
pass
finally:
pass
return _inner_call_method(method)
SOLUTION_1 = "nums = input() and input().split()\nprint(sum(map(int, nums)))"
SOLUTION_2 = "import sys\nprint(sum(map(int, sys.stdin.buffer.read().split()[1:])))"
print("using call_method_fixed on solution 1: (should print 6)")
call_method_fixed(lambda: exec(SOLUTION_1), "3\n1 2 3")
print("using call_method_fixed on solution 2: (should print 6)")
call_method_fixed(lambda: exec(SOLUTION_2), "3\n1 2 3")
print("using original call_method on solution 1: (should print 6)")
call_method(lambda: exec(SOLUTION_1), "3\n1 2 3")
print("using original call_method on solution 2: (should fail with AttributeError: '_io.StringIO' object has no attribute 'buffer'")
call_method(lambda: exec(SOLUTION_2), "3\n1 2 3")
```
## Expected behavior
The use of `sys.stdin.buffer.read()` should not cause an error.
## Version info
I am using lighteval v 0.13 and I installed it via uv.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.