GuyTevet / GuyTevet/motion-diffusion-model

We don't have to encode_text in each denoise timestep (but compute them at the beginning), do we?

Open
#151 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
4.1k
Forks
462
PR merge metrics
No merged PRs in 30d

Description

https://github.com/GuyTevet/motion-diffusion-model/blob/8139dda55d90a58aa5a257ebf159b2ecfb78c632/model/mdm.py#L151C8-L151C8
```
class MDM(nn.Module):
......
def forward(self, x, timesteps, y=None):
"""
x: [batch_size, njoints, nfeats, max_frames], denoted x_t in the paper
timesteps: [batch_size] (int)
"""
bs, njoints, nfeats, nframes = x.shape
emb = self.embed_timestep(timesteps) # [1, bs, d]

force_mask = y.get('uncond', False)
if 'text' in self.cond_mode:
enc_text = self.encode_text(y['text'])
emb += self.embed_text(self.mask_cond(enc_text, force_mask=force_mask))
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in model/mdm.py at MDM.forward and inspect the denoising caller to determine whether the same text is passed across timesteps. Verify how encode_text and the conditioning mask are used before deciding where embeddings could be reused. Done means text conditioning is computed once per sampling run without changing model outputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.