huggingface / huggingface/evaluation-guidebook

[Contribution Proposal] New section: Evaluating LLMs on low-resource languages and non-Western cultural contexts

Open
#40 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.1k
Forks
125
PR merge metrics
No merged PRs in 30d

Description

Hi Clémentine and team,

I'm an independent AI researcher with a background in public health, linguistics, and multilingual editing. I write in French, English, and Haitian Creole, and I research AI evaluation methodology.

I'd like to propose and write a new section for the guidebook covering a significant gap: how to evaluate LLMs on low-resource languages and non-Western cultural contexts.
The current guidebook covers evaluation methodology comprehensively for English and high-resource languages. But researchers working
with Haitian Creole, Caribbean French, and similar contexts face specific challenges not currently addressed:

1. Most benchmarks assume Western cultural references as ground truth a model "failing" may reflect cultural mismatch,
not capability failure
2. Low-resource languages have no standardized evaluation sets. Practitioners have no guidance on building one from scratch
3. Native speaker validation differs from crowdsourced annotation. No best practices exist for small annotator pools

Proposed section structure:

## Evaluating LLMs on low-resource languages
### Why standard benchmarks fail for low-resource languages
### How to build a minimal evaluation set from scratch
### Native speaker validation: practical guidelines
### Cultural bias vs. capability failure: how to distinguish them
### Existing multilingual resources and their limitations
### Case study: Haitian Creole evaluation set

I can write this section in full. I only need:
- Confirmation this fits the guidebook scope
- Guidance on which folder to place the .md file

This connects to ongoing independent research on prompt stability
across linguistic variants (Reasoning Stability Index / RSI).

Independent AI Researcher
GitHub: https://github.com/fabthebest

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.