ByteDance-Seed / ByteDance-Seed/Bagel
Request for the Release of Complete Evaluation Codes for MMMU_CoT
- Dominant language
- Python
- Stars
- 6.2k
- Forks
- 545
- PR merge metrics
- No merged PRs in 30d
Description
In the README, it's mentioned that "For MMMU, we use CoT in the report, which improves the accuracy by about 2%. For evaluation of the open - ended answers, we use GPT - 4o for judgement." However, during my practical testing, I noticed that when using the `Bagel/eval/vlm/eval/mmmu/evaluate_mmmu_cot.py script,` I haven't been able to obtain any metric results. After further testing, it seems that `main_eval_only.py` can only perform basic calculations on the results of `evaluate_mmmu.py.`
I was wondering if it would be possible to release the corresponding metric calculation code. This would greatly assist me in accurately computing the metrics. Additionally, I couldn't locate the part related to GPT - 4o in the code. I'm curious to know if it might be feasible to make the complete MMMU evaluation code available. Thank you for your consideration.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the README's MMMU evaluation description and compare Bagel/eval/vlm/eval/mmmu/evaluate_mmmu_cot.py with main_eval_only.py. Trace where metric results and the GPT-4o judging step are expected to be produced. Done means the complete MMMU CoT evaluation path, including metric calculation and GPT-4o-related evaluation code, is available or clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100