v1.5版本的7B模型在element_ocr场景下大幅低于v1版本的2B模型,是否符合预期
- Dominant language
- Python
- Stars
- 11.5k
- Forks
- 877
- PR merge metrics
- No merged PRs in 30d
Description
使用https://github.com/VisualWebBench/VisualWebBench
进行评估,结果如下,其中element_ocr这个子任务,2B模型反而好很多,因为论文中没有展示各个子任务的精度,想确认下是否符合预期
2B模型
Model: , Task: web_caption, Scores: rouge_1: 4.54, rouge_2: 1.10, rouge_l: 4.15
Model: , Task: heading_ocr, Scores: rouge_1: 68.94, rouge_2: 67.12, rouge_l: 68.94
Model: , Task: element_ocr, Scores: rouge_1: 94.35, rouge_2: 93.10, rouge_l: 94.35
Model: , Task: action_prediction, Scores: accuracy: 5.34
Model: , Task: element_ground, Scores: accuracy: 93.70
7B模型
Model: , Task: web_caption, Scores: rouge_1: 25.71, rouge_2: 7.17, rouge_l: 23.22
Model: , Task: heading_ocr, Scores: rouge_1: 72.27, rouge_2: 68.17, rouge_l: 72.27
Model: , Task: element_ocr, Scores: rouge_1: 78.47, rouge_2: 75.70, rouge_l: 78.19
Model: , Task: action_prediction, Scores: accuracy: 16.73
Model: , Task: element_ground, Scores: accuracy: 93.70

Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported VisualWebBench evaluation for the v1.5 7B and v1 2B models, focusing on the element_ocr task and its ROUGE scores. Compare the model, checkpoint, and evaluation settings used for both runs. Done means determining whether the discrepancy is expected or identifying a reproducible evaluation or model issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100