bytedance / bytedance/UI-TARS

Questions on Action Space Unification and Dataset Splitting in UI-TARS

Open
#86 2 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.5k
Forks
877
PR merge metrics
No merged PRs in 30d

Description

In the paper, it is mentioned that UI-TARS integrates multiple existing datasets (such as MM-Mind2Web, GUIAct, AITW, AITZ, AndroidControl, GUI-Odyssey, AMEX, etc.) and standardizes their action spaces into a unified format. I have two questions regarding this process:

1. **Action Space Unification Method**:
- How exactly are the action representations from these different datasets unified into UI-TARS's action space? For example, different datasets may have different action definitions and parameter formats. How does UI-TARS handle these differences?
- Is there a detailed mapping rule or code snippet that can be referenced?

2. **Dataset Splitting**:
- During the training of UI-TARS, were the test set data from these open-source datasets also included in the training set? If so, this could potentially affect the model's evaluation results.
- Can you clarify how the training and test sets of these datasets are split, and whether test set data was used during training?

**Suggestions**:
- Provide a detailed document or code snippet explaining how the action spaces from different datasets are unified into UI-TARS's action space.
- Clarify the dataset splitting situation, especially whether test set data was used in the training process.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the paper's description of UI-TARS action-space unification and dataset composition, then inspect the repository's training and dataset-preparation entry points. Document the mapping for each listed dataset and state whether training includes any test data, with sources or code references supporting the split policy.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.