bytedance / bytedance/UI-TARS

Confuse about ActionSpace for MobileUse

Open
#122 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.5k
Forks
877
PR merge metrics
No merged PRs in 30d

Description

Action space in [README_v1.md](https://github.com/bytedance/UI-TARS/blob/main/README_v1.md) for Mobile is:
```
click(start_box='<|box_start|>(x1,y1)<|box_end|>')
long_press(start_box='<|box_start|>(x1,y1)<|box_end|>', time='')
type(content='')
scroll(start_box='<|box_start|>(x1,y1)<|box_end|>', end_box='<|box_start|>(x3,y3)<|box_end|>')
press_home()
press_back()
finished(content='') # Submit the task regardless of whether it succeeds or fails.
```

And in [prompts.py](https://github.com/bytedance/UI-TARS/blob/main/prompts.py) for Mobile is:
```
click(start_box='<|box_start|>(x1,y1)<|box_end|>')
long_press(start_box='<|box_start|>(x1,y1)<|box_end|>')
type(content='') #If you want to submit your input, use "\\n" at the end of `content`.
scroll(start_box='<|box_start|>(x1,y1)<|box_end|>', direction='down or up or right or left')
open_app(app_name=\'\')
drag(start_box='<|box_start|>(x1,y1)<|box_end|>', end_box='<|box_start|>(x3,y3)<|box_end|>')
press_home()
press_back()
finished(content='xxx') # Use escape characters \\', \\", and \\n in content part to ensure we can parse the content in normal python string format.
```

Confused about action `scroll` and `drag` in `prompt.py`, cause these two actions are for compter in previous `README_v1.md`.

Is that a mistake?

Contributor guide

No contributing guide indexed for this repository

Research direction

Compare the Mobile action definitions in README_v1.md with those in prompts.py, starting with the surrounding prompt construction and any nearby platform-specific definitions. Determine which action set is intended, then reconcile the README and prompt so their Mobile actions and parameters agree; done means the discrepancy is no longer present.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.