lightly-ai / lightly-ai/lightly-train

[FEAT] DEIMv2 implementation

Open
#756 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
1.7k
Forks
116
Avg merge
2d 21h
Merged PRs (30d)
6

Description

### 💡 Is your feature request related to a problem?

I would like to integrate DEIMv2 into lightly-train. DEIMv2 has 2 families of models,
**DINOv3** and **HGNetv2/LiteEncoder**, but I would suggest only DINOv3-based for now.

### 🧰 Describe the solution you'd like

A lot of code is already in place, though some will need divergence resolution.
**DEIMCriterion**(looks close, but differs from **DFINECriterion** with `mal_alpha=None`,
`use_uni_set=True` and loss `loss_mal` added) and **DEIMTransformer** will need to be added.

### 🛠 Alternatives you've considered

Not sure, if applicable

### 📝 Additional context
Some of code with divergence:
- when class LQE creates MLP, lightly just uses default "relu", while DEIM allows to create MLP with a different one.
- box_cxcywh_to_xyxy function in DEIM adds clamp(min=0.0) to width and height
- layer name converter will be required when loading checkpoints
- HybridEncoder:
- fusion of upsample_feat+feat_low and downsample_feat+feat_height in DEIMv2 can be via both sum and concatenation,
while Lighly always go for concatenation. And DEIMv2 pretrained models with sum
- DEIMv2 has larger encoder block variety covering RT-DETR, D-FINE and DEIM aggregation path.
- DEIMv2 uses 1x1 conv + depthwise spatial stride-2 conv for downsampling in both D-FINE and DEIM
- DEIMv2 does not always uses input projection and goes for nn.Identity when in_channels is equal to hidden_dim, which is the case for DINOv3
- DEIMv2 has additional parameters and imports in HungarianMatcher to accomodate late stage training, where it switches to class-score × IoU ranking objective.
- DEIMCriterion can inherit from DFINECriterion to add parameters and loss.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing DFINECriterion, HybridEncoder, HungarianMatcher, LQE MLP, box_cxcywh_to_xyxy, and checkpoint layer-name conversion code. The work is done when the DINOv3-based DEIMv2 integration includes DEIMCriterion and DEIMTransformer and accounts for the listed model, loss, checkpoint, encoder, and matcher divergences.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.