lightly-ai / lightly-ai/lightly

Support gradient checkpointing in ViT ImageNet benchmarks

Open
#2,054 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement help wanted
Dominant language
Python
Stars
3.8k
Forks
367
Avg merge
3d 22h
Merged PRs (30d)
5

Description

`MaskedVisionTransformerTIMM.encode()` calls `self.vit.blocks(tokens)` directly (`lightly/models/modules/masked_vision_transformer_timm.py:131`), so timm's `vit.set_grad_checkpointing(True)` has no effect: timm reads that flag only inside `forward_features`, which the wrapper bypasses. There is currently no way to trade compute for memory on the ViT-B/16 benchmarks.

**Proposal:**
- [ ] Add an opt-in flag on `MaskedVisionTransformerTIMM` that routes `encode()` and `forward_intermediates()` through timm's `checkpoint_seq` / `torch.utils.checkpoint` when set. Follow the existing idiom in `masked_causal_vision_transformer.py:204`.
- [ ] Add `--grad-checkpointing` to `benchmarks/imagenet/vitb16/main.py`, enabling it by walking `model.modules()`.
- [ ] Test that checkpointed output matches the plain path and gradients flow (`tests/models/`).

**Coverage:** the wrapper change covers the 6 methods that use `MaskedVisionTransformerTIMM` (capi, dino, dinov2, ibot, mae, pixio). aim (`MaskedCausalVisionTransformer`, already supports it) and lejepa/visreg (raw timm ViT, native flag works) need one extra branch in the walk. Decoder checkpointing for mae/pixio is a separate follow-up.

**Notes:**
- Shared module used by every ViT method, so this is a library change, not benchmark-only.
- ResNet50 is out of scope: torchvision ResNet has no built-in support and grad checkpointing buys little there.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with lightly/models/modules/masked_vision_transformer_timm.py:131 and the gradient-checkpointing idiom in masked_causal_vision_transformer.py:204. Update benchmarks/imagenet/vitb16/main.py and inspect tests/models/ for the relevant model tests. Done means the opt-in path and --grad-checkpointing flag work across the listed methods, checkpointed outputs match the plain path, and gradients flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.