ByteDance-Seed / ByteDance-Seed/Bagel

Timestep Bias and Embedder Performance

Open
#199 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
6.2k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

Regarding the timestep, I noticed that during the sampling of the dataset, the timestep is set as `timestep = np.random.randn()`. Then, during training, it is updated as:
`packed_timesteps = torch.sigmoid(packed_timesteps)`
Doesn't this mean that during training, most of the time is spent on timesteps closer to the middle of the range?

Additionally, I found that the time embedder is implemented as:
```
# --------------------------------------------------------
# TimestepEmbedder
# Reference:
# DiT: https://github.com/facebookresearch/DiT/blob/main/models.py
# --------------------------------------------------------
class TimestepEmbedder(nn.Module):
```
In the TimestepEmbedder, without modification, the output **t_freq** for **0 < t < 1** is extremely similar (although the subsequent MLP slightly separates them). I would like to discuss whether this could affect the performance.

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the timestep sampling and training snippets and the TimestepEmbedder class referenced in the issue. Compare the resulting timestep distribution and embedding separation, then determine whether this affects model performance; done means documenting a decision and, if warranted, a scoped change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.