EleutherAI / EleutherAI/lm-evaluation-harness
Incorrect Multilingual arc implementation
- Dominant language
- Python
- Stars
- 14k
- Forks
- 3.6k
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 35
Description
The okapi translated arc uses preprocessing for no reason (likely oversight as the preprocessing if for hellaswag). I am not sure if it destroys some instructions, but I don't think there is any preprocessing needed.
Line in question:
https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/okapi/arc_multilingual/utils.py#L6
Contributor guide
Research direction
Open lm_eval/tasks/okapi/arc_multilingual/utils.py at line 6 and inspect how its preprocessing is applied. Compare that use with the hellaswag preprocessing mentioned in the issue, then run the relevant repository checks or evaluation for the multilingual ARC task. Done means ARC no longer applies unnecessary preprocessing and the task still evaluates correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100