Lightning-AI / Lightning-AI/pytorch-lightning

Support mosaic optimizations as plugins

Open
#12,360 6 comments 9 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

3rd party feature performance
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

This library [mosaic](https://github.com/mosaicml/composer) has neat tricks for optimizing the models for faster training.
Each application is done as a single line to the model
```python
import composer.functional as cf
from torchvision import models

my_model = models.resnet18()

# add blurpool and squeeze excite layers
model = cf.apply_blurpool(my_model)
model = cf.apply_squeeze_excite(my_model)

# your own training code starts here
```

Which is something we can automatically do for users under the hood if they want to enable the mosaic optimizations.

I propose an API like this
```python
import pytorch_lightning as pl

trainer = pl.Trainer(plugins=[
mosaic.BlurPool(replace_convs=True, replace_maxpools=True, blur_first=True),
mosaic.ChannelsLast(),
mosaic.CutMix(num_classes=10),
mosaic.LabelSmoothing(smoothing=0.1),
])

```

cc @borda @akihironitta @Borda @carmocca @tchaton

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No implementation files or tests are identified; start by reviewing the Trainer plugins entry point and the composer.functional optimization calls shown in the issue. Done means the listed Mosaic optimizations can be enabled through the proposed plugin API with the requested configuration options.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
developer-experience, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.