huggingface / huggingface/lighteval

hellaswag_arabic_pfn uses eval() instead of ast.literal_eval() on a dataset field, arbitrary code execution from an untrusted HF dataset

Open Beginner friendly
#1,293 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

### Bug Description

`hellaswag_arabic_pfn` in `src/lighteval/tasks/multilingual/tasks/arabic.py` (the prompt function for the `hellaswag_okapi_ar` task) parses a dataset field with the builtin `eval()` instead of `ast.literal_eval()`:

```python
def hellaswag_arabic_pfn(line, task_name: str = None):
ctx = re.sub(r"\[.*?\]", "", line["ctx"])
endings = [
re.sub(r"\[.*?\]", "", e) for e in eval(line["endings"])
] # endings is a string representation of a list
```

`line["endings"]` comes directly from the `hf_repo="OALL/AlGhafa-Arabic-LLM-Benchmark-Translated"` dataset on the Hub, a community-hosted dataset outside lighteval's own control. Since `eval()` executes arbitrary Python, not just list literals, a crafted `endings` field in that dataset (or any future revision of it, or a similarly-named dataset someone points a custom task at) can run arbitrary code the moment this task is loaded and evaluated, i.e. on whatever machine is running the benchmark.

### Steps to Reproduce

```python
import ast

legit = "['ending one', 'ending two', 'ending three']"
print(eval(legit)) # ['ending one', 'ending two', 'ending three']
print(ast.literal_eval(legit)) # same result, safely

malicious = "__import__('os').system('id > /tmp/pwned_by_dataset.txt')"
eval(malicious) # executes the shell command, file gets created
```

Confirmed locally: `eval()` on the malicious payload actually ran the injected shell command and wrote the file, while `ast.literal_eval()` on the exact same payload safely raised `ValueError: malformed node or string` instead of executing anything, and still returns the identical result for the legitimate case.

### Expected Behavior

This should use `ast.literal_eval()`, which is in fact the pattern lighteval already uses everywhere else for this exact situation, string-encoded list fields coming from HF datasets:

- `src/lighteval/tasks/tasks/sacrebleu.py:39` — `line["translation"] = ast.literal_eval(line["translation"])`
- `src/lighteval/tasks/tasks/race_high.py:33` — `line["problems"] = ast.literal_eval(line["problems"])`
- `src/lighteval/tasks/tasks/musr.py:39` — `choices = ast.literal_eval(line["choices"])`
- `src/lighteval/tasks/multilingual/tasks/swiss_legal/main.py:366` — same pattern

`arabic.py:603` is the one place using raw `eval()` for what looks like the identical use case, so this looks like a one-off inconsistency rather than an intentional choice, and the fix is a straightforward drop-in replacement (`ast.literal_eval` handles list/string/number literals identically to `eval` for well-formed input, it just refuses anything that isn't a literal).

### Affected code

`src/lighteval/tasks/multilingual/tasks/arabic.py`, line 603, inside `hellaswag_arabic_pfn`.

### System

huggingface/lighteval main branch.

Contributor guide

No contributing guide indexed for this repository

Research direction

Open src/lighteval/tasks/multilingual/tasks/arabic.py and inspect hellaswag_arabic_pfn around line 603, then compare the existing ast.literal_eval patterns in the referenced task files. Done means legitimate string-encoded endings still parse correctly while the malicious payload is rejected without executing code.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
security
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.