bytedance / bytedance/UI-TARS

v1.5版本的7B模型在element_ocr场景下大幅低于v1版本的2B模型,是否符合预期

Open
#156 16 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.5k
Forks
877
PR merge metrics
No merged PRs in 30d

Description

使用https://github.com/VisualWebBench/VisualWebBench

进行评估,结果如下,其中element_ocr这个子任务,2B模型反而好很多,因为论文中没有展示各个子任务的精度,想确认下是否符合预期

2B模型
Model: , Task: web_caption, Scores: rouge_1: 4.54, rouge_2: 1.10, rouge_l: 4.15
Model: , Task: heading_ocr, Scores: rouge_1: 68.94, rouge_2: 67.12, rouge_l: 68.94
Model: , Task: element_ocr, Scores: rouge_1: 94.35, rouge_2: 93.10, rouge_l: 94.35
Model: , Task: action_prediction, Scores: accuracy: 5.34
Model: , Task: element_ground, Scores: accuracy: 93.70

7B模型
Model: , Task: web_caption, Scores: rouge_1: 25.71, rouge_2: 7.17, rouge_l: 23.22
Model: , Task: heading_ocr, Scores: rouge_1: 72.27, rouge_2: 68.17, rouge_l: 72.27
Model: , Task: element_ocr, Scores: rouge_1: 78.47, rouge_2: 75.70, rouge_l: 78.19
Model: , Task: action_prediction, Scores: accuracy: 16.73
Model: , Task: element_ground, Scores: accuracy: 93.70

![Image](https://github.com/user-attachments/assets/ed32da54-a694-4996-b42d-bdb61f99b200)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported VisualWebBench evaluation for the v1.5 7B and v1 2B models, focusing on the element_ocr task and its ROUGE scores. Compare the model, checkpoint, and evaluation settings used for both runs. Done means determining whether the discrepancy is expected or identifying a reproducible evaluation or model issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.