ByteDance-Seed / ByteDance-Seed/Bagel

Inconsistent KRIS-Bench Results: Unable to Reproduce Reported Scores Using Official Evaluation Script

Open
#280 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.2k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

Thank you for providing such an impressive model and for sharing the evaluation details of KRIS-Bench.

When attempting to reproduce your reported results using the same hyperparameters (`cfg_text_scale=4, cfg_img_scale=1.5, cfg_renorm_min=0`), I consistently obtained significantly higher scores than those shown in your reference: https://github.com/mercurystraw/Kris_Bench/issues/2#issue-3147374124. Even when I evaluated using the inference outputs you provided in https://huggingface.co/datasets/Andy1621/BAGEL_results, the scores remained noticeably higher than the ones you originally published.

To avoid any discrepancies introduced by internal tooling, I used the official KRIS-Bench evaluation code. The only change I made was updating the model API from "gpt-4o" to "gpt-4o-2024-11-20".

Could you help verify whether the previously reported results can be reproduced using the official evaluation script?

For reference, my evaluation outputs are as follows:

Using my own inference results (not supporting multi-image input):
```
Attribute Perception:
VC: 81.64
VQ: 82.36
IF: 47.64
AVG: 70.55
Spatial Perception:
VC: 76.75
VQ: 92.00
IF: 39.00
AVG: 69.25
Temporal Prediction:
VC: 0.00
VQ: 0.00
IF: 0.00
AVG: 0.00
Social Science:
VC: 79.00
VQ: 86.00
IF: 43.01
KP: 33.60
AVG: 63.65
Natural Science:
VC: 75.13
VQ: 91.16
IF: 44.21
KP: 32.98
AVG: 64.42
Logical Reasoning:
VC: 85.67
VQ: 91.95
IF: 30.97
KP: 26.11
AVG: 62.86
Instruction Decomposition:
VC: 34.00
VQ: 36.33
IF: 41.17
AVG: 37.17
Factual Knowledge:
AVG: 58.63
Conceptual Knowledge:
AVG: 64.23
Procedural Knowledge:
AVG: 51.00
Overall:
AVG: 59.26
```

Using your released inference results:
```
Attribute Perception:
VC: 81.09
VQ: 82.73
IF: 48.82
AVG: 70.88
Spatial Perception:
VC: 80.25
VQ: 90.25
IF: 42.75
AVG: 71.08
Temporal Prediction:
VC: 37.84
VQ: 70.61
IF: 31.76
AVG: 46.73
Social Science:
VC: 84.60
VQ: 86.20
IF: 40.38
KP: 33.24
AVG: 64.93
Natural Science:
VC: 75.57
VQ: 90.84
IF: 39.45
KP: 29.61
AVG: 62.86
Logical Reasoning:
VC: 83.83
VQ: 94.83
IF: 31.67
KP: 23.96
AVG: 61.99
Instruction Decomposition:
VC: 35.17
VQ: 36.00
IF: 40.33
AVG: 37.17
Factual Knowledge:
AVG: 66.95
Conceptual Knowledge:
AVG: 63.36
Procedural Knowledge:
AVG: 50.71
Overall:
AVG: 61.49
```

Thank you for looking into this. If you have any questions, please feel free to let me know.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the official KRIS-Bench evaluation code and compare its results for the reported hyperparameters against both the author's inference outputs and the released BAGEL_results dataset. Check the gpt-4o versus gpt-4o-2024-11-20 API change and determine whether the previously reported scores can be reproduced; document the cause of any discrepancy.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.