OpenBMB / OpenBMB/MiniCPM

Request official evaluation scripts/configs for reproducing MiniCPM5-2B benchmark results

Open
#376 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
11.1k
Forks
763
Avg merge
6h 3m
Merged PRs (30d)
5

Description

Hello MiniCPM Team,

I am trying to reproduce the benchmark results of MiniCPM5-2B for academic research.

I have successfully deployed the released MiniCPM5-2B model following the official instructions. However, I could not find the complete evaluation pipeline used to obtain the benchmark results reported in the model card.

Could you please share the official evaluation scripts and configurations, including:

  1. Benchmark evaluation scripts (MATH-500, MMLU/MMLU-Pro, AIME, HMMT, etc.);
  2. Inference settings:
    • prompt/chat template;
    • whether enable_thinking=True is used;
    • generation parameters (max_new_tokens, temperature, top_p, do_sample, stop criteria, etc.);
  3. Answer extraction and grading scripts;
  4. Dataset preprocessing and evaluation commands.

I especially want to reproduce the reported MATH-500 score (94.6), and I would like to confirm whether the official evaluation uses any reasoning budget, length control, or special stopping strategy for the thinking mode.

Any scripts or configuration files would be greatly appreciated.

Thank you very much.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named in the issue. Start by locating any existing evaluation or benchmark-related material in the repository, then determine whether the requested MATH-500, MMLU/MMLU-Pro, AIME, and HMMT scripts, configurations, preprocessing, and grading commands can be added; done means the official reproduction workflow is available and documented.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.