OpenEuroLLM / OpenEuroLLM/Taskboard
Need provide details on reproducing belebele acc_norm scores
@vitiugin is already working on this.
Since Oct 24, 2025.
- Dominant language
- No language data
- Stars
- 3
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
My student Nam Luu Hoang and then also Koen Ooestermeijer were failing to reproduce the belebele scores for the 2B reference models (https://openeurollm.eu/blog/hplt-oellm-38-reference-models).
Resolved by switching to Cloze format (CF) evaluation, not MCF (multiple-choice format), but this should be documented.
Also, it seems that this works only in lighteval. Someone mentioned in Barcelona today that they fixed LM eval harness to get good belebele scores but Nam Luu failed with LM eval harness. So maybe the fix was only concerning the strange semicolon character (actually some Unicode garbage instead of it) that was reported, not the actual CF
Fedor, could you please find a good place where to document the exact way you got the scores and document it there?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.