Lightning-AI / Lightning-AI/lightning-thunder
Implement GroupNorm to invoke APEX GroupNorm for NeMo Stable Diffusion AutoEncoder performance
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
### 🐛 Bug
Applying `thunder.jit` to the `AutoEncoder` stage in NeMo's Stable Diffusion is slower than the eager mode (takes 0.198s per train step) where as `thunder.jit` 0.242s takes per step.
### To Reproduce
Steps to reproduce the behavior:
1. Apply the attached git diff to NeMo
[encoder.patch](https://github.com/Lightning-AI/lightning-thunder/files/15478010/encoder.patch)
2. Run NeMo using the command below:
```python
python examples/multimodal/text_to_image/stable_diffusion/sd_train.py trainer.precision=16 trainer.num_nodes=1 trainer.devices=1 ++exp_manager.max_time_per_run=00:00:03:00 trainer.max_steps=20 model.micro_batch_size=1 model.global_batch_size=1 model.data.synthetic_data=True exp_manager.exp_dir=/workspace/TestData/multimodal/stable_diffusion_train model.inductor=False model.cond_stage_config._target_=nemo.collections.multimodal.modules.stable_diffusion.encoders.modules.FrozenCLIPEmbedder ++model.cond_stage_config.version=openai/clip-vit-large-patch14 ++model.cond_stage_config.max_length=77 ~model.cond_stage_config.restore_from_path ~model.cond_stage_config.freeze ~model.cond_stage_config.layer model.unet_config.from_pretrained=null model.first_stage_config.from_pretrained=null model.unet_config.use_flash_attention=False model.unet_config.attention_resolutions=\[1\] model.unet_config.channel_mult=\[1\]
```
The last trace of the encode step can be found in the attached log.
[last_trace_SD_encoder.log](https://github.com/Lightning-AI/lightning-thunder/files/15477773/last_trace_SD_encoder.log)
### Solution
The `encoder.patch` attached also rewrites `GroupNorm` to `torch.nn.GroupNorm`, which ends up moving the operator from using APEX groupnorm to using eager's groupnorm. This is what causes the performance drop in stable diffusion.
We should be mapping GroupNorm via either mechanism to APEX's GroupNorm.
cc: @tfogal
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the attached encoder.patch and last_trace_SD_encoder.log, then inspect the GroupNorm mapping used when applying thunder.jit to the NeMo Stable Diffusion AutoEncoder. Reproduce the provided sd_train.py command and verify that GroupNorm uses APEX's implementation and that the Thunder step is no slower than eager mode.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100