bytedance / bytedance/UI-TARS

About Android world's reviews

Open
#121 5 comments 6 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.5k
Forks
877
PR merge metrics
No merged PRs in 30d

Description

Congratulations, your new model ui-tars 1.5 is a big improvement, and I'd like to reproduce the results of your paper on Android world, and let's ask you how you evaluate it, and what's the prompt like. My rating on the ui-tars-7b-dpo was only 20+, and I didn't improve on the ui-tars 1.5 7b model, wondering if it was a request formatting issue or something. Here's my request format, five pictures in total, four historical records, and all of them. I can give you advice on how you feel. Thank you very much.
"model": self.model,
"temperature": self.temperature,
"top-p": 1,
"top-k": 1,
"messages": [
{
"role": "system",
"content": [{"type": "text","text": "You are a helpful assistant."}]
},
{
"role": "user",
"content": [{"type": "text","text": "prompt + instruction"}]
},
{
"role": "assistant",
"content": "Thought: xxxx。 \n Action: xxxx)"
},
{
"role": "assistant",
"content": "Thought: xxxx。 \n Action: xxxx)"
},
..
..
..
..
{
"role": "user",
"content": [{"type": "image_url","image_url": {"url": "base64"}}]
},
{
"role": "assistant",
"content": "Thought: xxxx。 \n Action: xxxx)"
},
{
"role": "user",
"content": [{"type": "image_url","image_url": {"url": "base64"}}]
}
]
}
]

prompt=“You are a GUI agent. You are given a task and your action history, with screenshots. You need to perform the next action to complete the task.\n\n## Output Format\n\n```text\nThought: ...\nAction: ...\n```\n\n## Action Space\n\n- open_app(app_name='app_name', activity='MainActivity') # open an app with the name stored in the 'app_name' field, and 'activity'. This name should map to an packege name and activity which can be used to open the app using adb.\n\n- click(start_box='<|box_start|>(x1,y1)<|box_end|>') # click at a specific point on the screen using the coordinates '(x1,y1)'.\n\n- long_press(start_box='<|box_start|>(x1,y1)<|box_end|>', time='') # long press at a specific point on the screen using the coordinates '(x1,y1)'.\n\n- type(content='') # types the text in the 'content' field. It does not perform a click or press enter at the end.\n\n- scroll( start_box='<|box_start|>(x1,y1)<|box_end|>', end_box='<|box_start|>(x2,y2)<|box_end|>') # scroll from point '( x1,y1)' to point '(x2,y2)'.\n\n- press_home() # go to the home screen.\n\n- press_back() # go back to the previous screen.\n\n- press_enter() # press `enter` on the keyboard.\n\n- wait(time='') # wait a set number of seconds.\n\n- finished(content='') # Submit the task regardless of whether it succeeds or fails. \n\n## Note\n\n- Use English in `Thought` part.\n\n- Write a small plan and finally summarize your next action (with its target element) in one sentence in `Thought` part.\n\n## Instruction \n”

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Begin with the supplied request format and Android World evaluation details; compare the message history, image inputs, sampling parameters, and prompt, then document whether a formatting difference explains the score or provide a reproducible cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, python
Domain
machine-learning, mobile-dev
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.