lightly-ai / lightly-ai/lightly
Support gradient checkpointing in ViT ImageNet benchmarks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.8k
- Forks
- 367
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 5
Description
`MaskedVisionTransformerTIMM.encode()` calls `self.vit.blocks(tokens)` directly (`lightly/models/modules/masked_vision_transformer_timm.py:131`), so timm's `vit.set_grad_checkpointing(True)` has no effect: timm reads that flag only inside `forward_features`, which the wrapper bypasses. There is currently no way to trade compute for memory on the ViT-B/16 benchmarks.
**Proposal:**
- [ ] Add an opt-in flag on `MaskedVisionTransformerTIMM` that routes `encode()` and `forward_intermediates()` through timm's `checkpoint_seq` / `torch.utils.checkpoint` when set. Follow the existing idiom in `masked_causal_vision_transformer.py:204`.
- [ ] Add `--grad-checkpointing` to `benchmarks/imagenet/vitb16/main.py`, enabling it by walking `model.modules()`.
- [ ] Test that checkpointed output matches the plain path and gradients flow (`tests/models/`).
**Coverage:** the wrapper change covers the 6 methods that use `MaskedVisionTransformerTIMM` (capi, dino, dinov2, ibot, mae, pixio). aim (`MaskedCausalVisionTransformer`, already supports it) and lejepa/visreg (raw timm ViT, native flag works) need one extra branch in the walk. Decoder checkpointing for mae/pixio is a separate follow-up.
**Notes:**
- Shared module used by every ViT method, so this is a library change, not benchmark-only.
- ResNet50 is out of scope: torchvision ResNet has no built-in support and grad checkpointing buys little there.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with lightly/models/modules/masked_vision_transformer_timm.py:131 and the gradient-checkpointing idiom in masked_causal_vision_transformer.py:204. Update benchmarks/imagenet/vitb16/main.py and inspect tests/models/ for the relevant model tests. Done means the opt-in path and --grad-checkpointing flag work across the listed methods, checkpointed outputs match the plain path, and gradients flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100