EleutherAI / EleutherAI/lm-evaluation-harness

Incorrect Multilingual arc implementation

Open
#2,000 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
14k
Forks
3.6k
Avg merge
4d 2h
Merged PRs (30d)
35

Description

The okapi translated arc uses preprocessing for no reason (likely oversight as the preprocessing if for hellaswag). I am not sure if it destroys some instructions, but I don't think there is any preprocessing needed.

Line in question:
https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/okapi/arc_multilingual/utils.py#L6

Contributor guide

Open the contributing guide

Research direction

Open lm_eval/tasks/okapi/arc_multilingual/utils.py at line 6 and inspect how its preprocessing is applied. Compare that use with the hellaswag preprocessing mentioned in the issue, then run the relevant repository checks or evaluation for the multilingual ARC task. Done means ARC no longer applies unnecessary preprocessing and the task still evaluates correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.