google-research / google-research/mseb
❓ Two Questions Regarding MSEB Test Set Definition and Chinese and English subdataset
- Dominant language
- Python
- Stars
- 66
- Forks
- 14
- Avg merge
- 21m
- Merged PRs (30d)
- 10
Description
We appreciate the continuous work on multilingual benchmarks. We have two key questions regarding the MSEB and the associated datasets.
1. MSEB Test Set Definition
Could you please confirm the exact test set composition for the MSEB leaderboard?
Is the official MSEB test dataset the complete google/svq dataset? (https://huggingface.co/datasets/google/svq)
Or will a separate, dedicated test split be released solely for MSEB evaluation?
A clear definition of the test boundary is essential for accurate reproducibility and benchmarking.
2. Status of Chinese Language Subdataset Release
Given that the existing google/svq dataset already covers 12 languages, we would like to confirm if a specific **Chinese** or **English** subdataset will still be released from the google/svq dataset for separate benchmarking.
Thank you for your clarification.
Contributor guide
Research direction
Start with the MSEB leaderboard definition and the linked google/svq dataset referenced in the issue. Determine whether MSEB uses the complete dataset or a dedicated test split, and whether separate Chinese and English subsets are planned. Done means documenting an authoritative answer to both questions.
Written by the indexing model from the issue text.
Assessment
- Domain
- data, machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100