huggingface / huggingface/lighteval

[BUG] FileNotFoundError with newest lighteval 0.11.0

Open
#988 2 comments 2 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
I receive a FileNotFoundError:

```
│ /home/jxm/anaconda3/lib/python3.12/site-packages/datasets/load.py:1033 in dataset_module_factory │
│ │
│ 1030 │ │ │ │ │ ) from None │
│ 1031 │ │ │ │ raise e1 from None │
│ 1032 │ else: │
│ ❱ 1033 │ │ raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_p │
│ 1034 │
│ 1035 │
│ 1036 def load_dataset_builder( │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: Couldn't find any data file at /home/jxm/lighteval/openai/gsm8k.
```
## To Reproduce
```bash
lighteval accelerate "model_name=gpt2" "lighteval|gsm8k|0|0"
```

## Expected behavior
I expect it to work and run evaluation.

## Version info
0.11.0 on linux with conda/uv pip

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.