lightly-ai / lightly-ai/lightly-train

[BUG] DINOv3 LT-DETR: -notpretrained backbone aliases broken or missing for most sizes

Open
#933 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug dinov3 documentation object-detection
Dominant language
Python
Stars
1.7k
Forks
116
Avg merge
2d 21h
Merged PRs (30d)
6

Description

### Describe the Bug

You can't reliably turn off backbone pretraining for DINOv3 in LT-DETR object detection (`train_object_detection`). DINOv2 already does this the right way, using dedicated `*-notpretrained` model names. DINOv3 doesn't.

There are 9 DINOv3 backbone sizes for LT-DETR. Only one (`vitt16`) has a `-notpretrained-ltdetr` name at all, and it doesn't work: `dinov3/vitt16-notpretrained-ltdetr` and `dinov3/vitt16-ltdetr` point to the exact same config class. So the "notpretrained" name still loads pretrained weights. The other 8 sizes (ConvNeXt-tiny/small/base/large, ViT-tiny+/small/base/large) have no `-notpretrained-ltdetr` name at all.

DINOv2 gets this right for its main sizes: `dinov2/vits14-notpretrained-ltdetr`, `vitb14-notpretrained-ltdetr`, `vitl14-notpretrained-ltdetr`, and `vitg14-notpretrained-ltdetr` each point to their own config class (e.g. `DINOv2ViTSmallNotPretrained`), which points to a real "no weights" backbone entry (`url=""` in `dinov2_vit_src/configs/__init__.py`). No pretrained weights get downloaded.

There is a workaround for DINOv3 today: pass `model_args={"backbone_args": {"weights": None, "pretrained": False}}` to `train_object_detection`. It works, but it's undocumented and easy to get wrong — passing only one of the two keys either silently loads pretrained weights anyway, or crashes with a `TypeError`. That's not a reasonable way to ask for "just give me a random-init backbone."

### Current State (DINOv3 LT-DETR aliases)

| Size | Pretrained alias | `-notpretrained-ltdetr` status |
|---|---|---|
| ConvNeXt-Tiny | `dinov3/convnext-tiny-ltdetr` | missing |
| ConvNeXt-Small | `dinov3/convnext-small-ltdetr` | missing |
| ConvNeXt-Base | `dinov3/convnext-base-ltdetr` | missing |
| ConvNeXt-Large | `dinov3/convnext-large-ltdetr` | missing |
| ViT-Tiny (vitt16) | `dinov3/vitt16-ltdetr` | **broken** — duplicate of pretrained alias, silently still loads pretrained weights |
| ViT-Tiny+ (vitt16plus) | `dinov3/vitt16plus-ltdetr` | missing |
| ViT-Small (vits16) | `dinov3/vits16-ltdetr` | missing |
| ViT-Base (vitb16) | `dinov3/vitb16-ltdetr` | missing |
| ViT-Large (vitl16) | `dinov3/vitl16-ltdetr` | missing |

Also worth noting: ViT-Huge+ (`vith16plus`) and ViT-7B (`vit7b16`) have no LT-DETR support at all yet, pretrained or not. That's not part of this bug, just flagging it.

### Root Cause / Where to Fix

Two places need changes, to match how DINOv2 already works:

1. **`src/lightly_train/_models/dinov3/dinov3_package.py`** (`DINOv3Package.MODEL_NAME_TO_INFO`): only `vitt16-notpretrained` and `vitt16plus-notpretrained` exist as "no weights" backbone entries (`functools.partial(backbones.dinov3_vit*, pretrained=False)`). We need to add the same kind of entry for `convnext-tiny/small/base/large`, `vits16`, `vitb16`, and `vitl16`.
2. **`src/lightly_train/_task_models/ltdetr_object_detection/config.py`**: needs one correctly-wired `*-notpretrained-ltdetr` config class per size, following the DINOv2 pattern (see `DINOv2ViTSmallNotPretrained`, `DINOv2ViTBaseNotPretrained`, `DINOv2ViTLargeNotPretrained`, `DINOv2ViTGiantNotPretrained`, around lines 1011-1098). Each class should point `backbone_name` at the new entry from step 1. The existing `dinov3/vitt16-notpretrained-ltdetr` also needs fixing — right now it points to the same class as the pretrained version instead of its own.

### Expected Behavior

Every DINOv3 LT-DETR backbone size should have a working `dinov3/-notpretrained-ltdetr` name that actually gives you a randomly-initialized backbone — the same way DINOv2 already works, with no extra `backbone_args` needed.

### Documentation

`docs/source/object_detection.md` lists all supported LT-DETR model names, but never mentions `notpretrained` for DINOv2 or DINOv3. So this option isn't discoverable without reading the source. Once the names are fixed/added, please also document them there.

### Minor Related Gap (DINOv2)

Found while comparing against DINOv2: `dinov2/vits14-noreg-ltdetr` (the legacy no-registers variant) has no `-notpretrained-ltdetr` version, even though the underlying `vits14-noreg-notpretrained` entry already exists and is unused. Easy to add alongside the DINOv3 fix.

### Additional Context

Found this while checking whether `pretrained: True` needs to be set explicitly to get a DINOv3-pretrained backbone in LT-DETR (it doesn't — that already works automatically and is unrelated to this bug). This issue is about the opposite case: reliably getting a backbone that is **not** pretrained.

If anything above is unclear, please leave a comment and I'll clarify.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read src/lightly_train/_models/dinov3/dinov3_package.py and src/lightly_train/_task_models/ltdetr_object_detection/config.py, using the DINOv2 not-pretrained classes as the wiring reference. Update the documented aliases in docs/source/object_detection.md, then verify every listed DINOv3 size and the DINOv2 no-registers variant resolves to a distinct no-weights configuration without extra backbone_args.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, documentation, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
66/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.