bytedance / bytedance/1d-tokenizer

about 'fake 2D shape'

Open
#16 1 comment 4 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.2k
Forks
70
PR merge metrics
No merged PRs in 30d

Description

Hello,

It's an impressive work. I've noticed that in modeling/blocks.py line 140-143:
```python
# fake 2D shape
latent_tokens = latent_tokens.reshape(batch_size, self.width, self.num_latent_tokens, 1)
latent_tokens = self.conv_out(latent_tokens)
latent_tokens = latent_tokens.reshape(batch_size, self.token_size, 1, self.num_latent_tokens)
```
It's wired why organize 32 tokens in this way. Looks like you extract elements `[0, :, : : 32]` in latent_tokens with init shape [B, 32, 1024] (L-32 model) as the channel dim in output tokens. Why not use `permute` instead? Hope for an answer.
Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with modeling/blocks.py lines 140-143 and trace the shape of latent_tokens before and after each reshape and conv_out call. Compare that flow with the proposed permute approach and document the reason for the current layout, including whether the two operations are behaviorally equivalent.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.