open-compass / open-compass/VLMEvalKit

Unable to reproduce InternVL3.5 GPT-OSS on MMMU

Open
#1,223 6 comments 0 reactions 1 assignee View on GitHub

@PhoenixZ810 is already working on this.

Since Sep 12, 2025.

Dominant language
Python
Stars
4.4k
Forks
768
Avg merge
1d 10h
Merged PRs (30d)
17

Description

Hi,

Thanks for including InternVL3.5 in the repo. However, I tried to evaluate InternVL3_5-GPT-OSS-20B-A4B-Preview-Thinking on MMMU_DEV_VAL, and the model keeps repeating itself at the end of the response. I've attached the result from this command python run.py --data MMMU_DEV_VAL --model InternVL3_5-GPT-OSS-20B-A4B-Preview-Thinking --verbose --reuse. I believe there might be some template adaptation that needs to be done.

Best,
Yu-Cheng

InternVL3_5-GPT-OSS-20B-A4B-Preview-Thinking_MMMU_DEV_VAL_openai_result.xlsx

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.