Questions on Action Space Unification and Dataset Splitting in UI-TARS
- Dominant language
- Python
- Stars
- 11.5k
- Forks
- 877
- PR merge metrics
- No merged PRs in 30d
Description
In the paper, it is mentioned that UI-TARS integrates multiple existing datasets (such as MM-Mind2Web, GUIAct, AITW, AITZ, AndroidControl, GUI-Odyssey, AMEX, etc.) and standardizes their action spaces into a unified format. I have two questions regarding this process:
1. **Action Space Unification Method**:
- How exactly are the action representations from these different datasets unified into UI-TARS's action space? For example, different datasets may have different action definitions and parameter formats. How does UI-TARS handle these differences?
- Is there a detailed mapping rule or code snippet that can be referenced?
2. **Dataset Splitting**:
- During the training of UI-TARS, were the test set data from these open-source datasets also included in the training set? If so, this could potentially affect the model's evaluation results.
- Can you clarify how the training and test sets of these datasets are split, and whether test set data was used during training?
**Suggestions**:
- Provide a detailed document or code snippet explaining how the action spaces from different datasets are unified into UI-TARS's action space.
- Clarify the dataset splitting situation, especially whether test set data was used in the training process.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the paper's description of UI-TARS action-space unification and dataset composition, then inspect the repository's training and dataset-preparation entry points. Document the mapping for each listed dataset and state whether training includes any test data, with sources or code references supporting the split policy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100