google-research / google-research/mseb

❓ Two Questions Regarding MSEB Test Set Definition and Chinese and English subdataset

Open
#288 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
66
Forks
14
Avg merge
21m
Merged PRs (30d)
10

Description

We appreciate the continuous work on multilingual benchmarks. We have two key questions regarding the MSEB and the associated datasets.

1. MSEB Test Set Definition
Could you please confirm the exact test set composition for the MSEB leaderboard?
Is the official MSEB test dataset the complete google/svq dataset? (https://huggingface.co/datasets/google/svq)
Or will a separate, dedicated test split be released solely for MSEB evaluation?
A clear definition of the test boundary is essential for accurate reproducibility and benchmarking.

2. Status of Chinese Language Subdataset Release
Given that the existing google/svq dataset already covers 12 languages, we would like to confirm if a specific **Chinese** or **English** subdataset will still be released from the google/svq dataset for separate benchmarking.

Thank you for your clarification.

Contributor guide

Open the contributing guide

Research direction

Start with the MSEB leaderboard definition and the linked google/svq dataset referenced in the issue. Determine whether MSEB uses the complete dataset or a dedicated test split, and whether separate Chinese and English subsets are planned. Done means documenting an authoritative answer to both questions.

Written by the indexing model from the issue text.

Assessment

Domain
data, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.