Lightning-AI / Lightning-AI/lightning-thunder

Implement GroupNorm to invoke APEX GroupNorm for NeMo Stable Diffusion AutoEncoder performance

Open
#468 15 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug nemo performance
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

### 🐛 Bug

Applying `thunder.jit` to the `AutoEncoder` stage in NeMo's Stable Diffusion is slower than the eager mode (takes 0.198s per train step) where as `thunder.jit` 0.242s takes per step.

### To Reproduce

Steps to reproduce the behavior:

1. Apply the attached git diff to NeMo
[encoder.patch](https://github.com/Lightning-AI/lightning-thunder/files/15478010/encoder.patch)

2. Run NeMo using the command below:
```python
python examples/multimodal/text_to_image/stable_diffusion/sd_train.py trainer.precision=16 trainer.num_nodes=1 trainer.devices=1 ++exp_manager.max_time_per_run=00:00:03:00 trainer.max_steps=20 model.micro_batch_size=1 model.global_batch_size=1 model.data.synthetic_data=True exp_manager.exp_dir=/workspace/TestData/multimodal/stable_diffusion_train model.inductor=False model.cond_stage_config._target_=nemo.collections.multimodal.modules.stable_diffusion.encoders.modules.FrozenCLIPEmbedder ++model.cond_stage_config.version=openai/clip-vit-large-patch14 ++model.cond_stage_config.max_length=77 ~model.cond_stage_config.restore_from_path ~model.cond_stage_config.freeze ~model.cond_stage_config.layer model.unet_config.from_pretrained=null model.first_stage_config.from_pretrained=null model.unet_config.use_flash_attention=False model.unet_config.attention_resolutions=\[1\] model.unet_config.channel_mult=\[1\]
```
The last trace of the encode step can be found in the attached log.
[last_trace_SD_encoder.log](https://github.com/Lightning-AI/lightning-thunder/files/15477773/last_trace_SD_encoder.log)

### Solution

The `encoder.patch` attached also rewrites `GroupNorm` to `torch.nn.GroupNorm`, which ends up moving the operator from using APEX groupnorm to using eager's groupnorm. This is what causes the performance drop in stable diffusion.

We should be mapping GroupNorm via either mechanism to APEX's GroupNorm.

cc: @tfogal

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached encoder.patch and last_trace_SD_encoder.log, then inspect the GroupNorm mapping used when applying thunder.jit to the NeMo Stable Diffusion AutoEncoder. Reproduce the provided sd_train.py command and verify that GroupNorm uses APEX's implementation and that the Thunder step is no slower than eager mode.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.