ByteDance-Seed / ByteDance-Seed/Bagel
Inconsistent KRIS-Bench Results: Unable to Reproduce Reported Scores Using Official Evaluation Script
- Dominant language
- Python
- Stars
- 6.2k
- Forks
- 545
- PR merge metrics
- No merged PRs in 30d
Description
Thank you for providing such an impressive model and for sharing the evaluation details of KRIS-Bench.
When attempting to reproduce your reported results using the same hyperparameters (`cfg_text_scale=4, cfg_img_scale=1.5, cfg_renorm_min=0`), I consistently obtained significantly higher scores than those shown in your reference: https://github.com/mercurystraw/Kris_Bench/issues/2#issue-3147374124. Even when I evaluated using the inference outputs you provided in https://huggingface.co/datasets/Andy1621/BAGEL_results, the scores remained noticeably higher than the ones you originally published.
To avoid any discrepancies introduced by internal tooling, I used the official KRIS-Bench evaluation code. The only change I made was updating the model API from "gpt-4o" to "gpt-4o-2024-11-20".
Could you help verify whether the previously reported results can be reproduced using the official evaluation script?
For reference, my evaluation outputs are as follows:
Using my own inference results (not supporting multi-image input):
```
Attribute Perception:
VC: 81.64
VQ: 82.36
IF: 47.64
AVG: 70.55
Spatial Perception:
VC: 76.75
VQ: 92.00
IF: 39.00
AVG: 69.25
Temporal Prediction:
VC: 0.00
VQ: 0.00
IF: 0.00
AVG: 0.00
Social Science:
VC: 79.00
VQ: 86.00
IF: 43.01
KP: 33.60
AVG: 63.65
Natural Science:
VC: 75.13
VQ: 91.16
IF: 44.21
KP: 32.98
AVG: 64.42
Logical Reasoning:
VC: 85.67
VQ: 91.95
IF: 30.97
KP: 26.11
AVG: 62.86
Instruction Decomposition:
VC: 34.00
VQ: 36.33
IF: 41.17
AVG: 37.17
Factual Knowledge:
AVG: 58.63
Conceptual Knowledge:
AVG: 64.23
Procedural Knowledge:
AVG: 51.00
Overall:
AVG: 59.26
```
Using your released inference results:
```
Attribute Perception:
VC: 81.09
VQ: 82.73
IF: 48.82
AVG: 70.88
Spatial Perception:
VC: 80.25
VQ: 90.25
IF: 42.75
AVG: 71.08
Temporal Prediction:
VC: 37.84
VQ: 70.61
IF: 31.76
AVG: 46.73
Social Science:
VC: 84.60
VQ: 86.20
IF: 40.38
KP: 33.24
AVG: 64.93
Natural Science:
VC: 75.57
VQ: 90.84
IF: 39.45
KP: 29.61
AVG: 62.86
Logical Reasoning:
VC: 83.83
VQ: 94.83
IF: 31.67
KP: 23.96
AVG: 61.99
Instruction Decomposition:
VC: 35.17
VQ: 36.00
IF: 40.33
AVG: 37.17
Factual Knowledge:
AVG: 66.95
Conceptual Knowledge:
AVG: 63.36
Procedural Knowledge:
AVG: 50.71
Overall:
AVG: 61.49
```
Thank you for looking into this. If you have any questions, please feel free to let me know.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the official KRIS-Bench evaluation code and compare its results for the reported hyperparameters against both the author's inference outputs and the released BAGEL_results dataset. Check the gpt-4o versus gpt-4o-2024-11-20 API change and determine whether the previously reported scores can be reproduced; document the cause of any discrepancy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100