bytedance / bytedance/1d-tokenizer

Ask detailed questions about the permutation strategy in the RAR

Open
#48 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.2k
Forks
70
PR merge metrics
No merged PRs in 30d

Description

Hello author, I am very grateful for your excellent work and generous open source. After reading your source code, I have a small question, why the permute strategy does not shuffle the initial condition? Anyway, you will return to the original permutation after annealing. Here are some of the source codes that I am confused about:

```python
# cls_token, condition, the permute does not impact these prefix tokens.
prefix = 2
pos_embed_prefix = pos_embed[:, :prefix]
pos_embed_postfix = self.shuffle(pos_embed[:, prefix:prefix+self.image_seq_len], orders)
```

Is it because of the contribution of target-aware positional embedding, it doesn't matter if it is not shuffled?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the permutation strategy around the shown prefix and postfix positional-embedding code, including self.shuffle and orders. Check how the RAR handles the cls token, condition, and image tokens during annealing. Done means documenting why the prefix tokens are excluded from shuffling and whether target-aware positional embeddings affect that choice.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.