ByteDance-Seed / ByteDance-Seed/Bagel

Request for the Release of Complete Evaluation Codes for MMMU_CoT

Open
#223 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.2k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

In the README, it's mentioned that "For MMMU, we use CoT in the report, which improves the accuracy by about 2%. For evaluation of the open - ended answers, we use GPT - 4o for judgement." However, during my practical testing, I noticed that when using the `Bagel/eval/vlm/eval/mmmu/evaluate_mmmu_cot.py script,` I haven't been able to obtain any metric results. After further testing, it seems that `main_eval_only.py` can only perform basic calculations on the results of `evaluate_mmmu.py.`

I was wondering if it would be possible to release the corresponding metric calculation code. This would greatly assist me in accurately computing the metrics. Additionally, I couldn't locate the part related to GPT - 4o in the code. I'm curious to know if it might be feasible to make the complete MMMU evaluation code available. Thank you for your consideration.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the README's MMMU evaluation description and compare Bagel/eval/vlm/eval/mmmu/evaluate_mmmu_cot.py with main_eval_only.py. Trace where metric results and the GPT-4o judging step are expected to be produced. Done means the complete MMMU CoT evaluation path, including metric calculation and GPT-4o-related evaluation code, is available or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.