ByteDance-Seed / ByteDance-Seed/Bagel
Timestep Bias and Embedder Performance
- Dominant language
- Python
- Stars
- 6.2k
- Forks
- 545
- PR merge metrics
- No merged PRs in 30d
Description
Regarding the timestep, I noticed that during the sampling of the dataset, the timestep is set as `timestep = np.random.randn()`. Then, during training, it is updated as:
`packed_timesteps = torch.sigmoid(packed_timesteps)`
Doesn't this mean that during training, most of the time is spent on timesteps closer to the middle of the range?
Additionally, I found that the time embedder is implemented as:
```
# --------------------------------------------------------
# TimestepEmbedder
# Reference:
# DiT: https://github.com/facebookresearch/DiT/blob/main/models.py
# --------------------------------------------------------
class TimestepEmbedder(nn.Module):
```
In the TimestepEmbedder, without modification, the output **t_freq** for **0 < t < 1** is extremely similar (although the subsequent MLP slightly separates them). I would like to discuss whether this could affect the performance.
Contributor guide
No contributing guide indexed for this repository
Research direction
Review the timestep sampling and training snippets and the TimestepEmbedder class referenced in the issue. Compare the resulting timestep distribution and embedding separation, then determine whether this affects model performance; done means documenting a decision and, if warranted, a scoped change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100