bytedance / bytedance/1d-tokenizer
Ask detailed questions about the permutation strategy in the RAR
- Dominant language
- Jupyter Notebook
- Stars
- 1.2k
- Forks
- 70
- PR merge metrics
- No merged PRs in 30d
Description
Hello author, I am very grateful for your excellent work and generous open source. After reading your source code, I have a small question, why the permute strategy does not shuffle the initial condition? Anyway, you will return to the original permutation after annealing. Here are some of the source codes that I am confused about:
```python
# cls_token, condition, the permute does not impact these prefix tokens.
prefix = 2
pos_embed_prefix = pos_embed[:, :prefix]
pos_embed_postfix = self.shuffle(pos_embed[:, prefix:prefix+self.image_seq_len], orders)
```
Is it because of the contribution of target-aware positional embedding, it doesn't matter if it is not shuffled?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the permutation strategy around the shown prefix and postfix positional-embedding code, including self.shuffle and orders. Check how the RAR handles the cls token, condition, and image tokens during annealing. Done means documenting why the prefix tokens are excluded from shuffling and whether target-aware positional embeddings affect that choice.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100