lightly-ai / lightly-ai/lightly-train
[BUG] DINOv3 LT-DETR: -notpretrained backbone aliases broken or missing for most sizes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 116
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 6
Description
### Describe the Bug
You can't reliably turn off backbone pretraining for DINOv3 in LT-DETR object detection (`train_object_detection`). DINOv2 already does this the right way, using dedicated `*-notpretrained` model names. DINOv3 doesn't.
There are 9 DINOv3 backbone sizes for LT-DETR. Only one (`vitt16`) has a `-notpretrained-ltdetr` name at all, and it doesn't work: `dinov3/vitt16-notpretrained-ltdetr` and `dinov3/vitt16-ltdetr` point to the exact same config class. So the "notpretrained" name still loads pretrained weights. The other 8 sizes (ConvNeXt-tiny/small/base/large, ViT-tiny+/small/base/large) have no `-notpretrained-ltdetr` name at all.
DINOv2 gets this right for its main sizes: `dinov2/vits14-notpretrained-ltdetr`, `vitb14-notpretrained-ltdetr`, `vitl14-notpretrained-ltdetr`, and `vitg14-notpretrained-ltdetr` each point to their own config class (e.g. `DINOv2ViTSmallNotPretrained`), which points to a real "no weights" backbone entry (`url=""` in `dinov2_vit_src/configs/__init__.py`). No pretrained weights get downloaded.
There is a workaround for DINOv3 today: pass `model_args={"backbone_args": {"weights": None, "pretrained": False}}` to `train_object_detection`. It works, but it's undocumented and easy to get wrong — passing only one of the two keys either silently loads pretrained weights anyway, or crashes with a `TypeError`. That's not a reasonable way to ask for "just give me a random-init backbone."
### Current State (DINOv3 LT-DETR aliases)
| Size | Pretrained alias | `-notpretrained-ltdetr` status |
|---|---|---|
| ConvNeXt-Tiny | `dinov3/convnext-tiny-ltdetr` | missing |
| ConvNeXt-Small | `dinov3/convnext-small-ltdetr` | missing |
| ConvNeXt-Base | `dinov3/convnext-base-ltdetr` | missing |
| ConvNeXt-Large | `dinov3/convnext-large-ltdetr` | missing |
| ViT-Tiny (vitt16) | `dinov3/vitt16-ltdetr` | **broken** — duplicate of pretrained alias, silently still loads pretrained weights |
| ViT-Tiny+ (vitt16plus) | `dinov3/vitt16plus-ltdetr` | missing |
| ViT-Small (vits16) | `dinov3/vits16-ltdetr` | missing |
| ViT-Base (vitb16) | `dinov3/vitb16-ltdetr` | missing |
| ViT-Large (vitl16) | `dinov3/vitl16-ltdetr` | missing |
Also worth noting: ViT-Huge+ (`vith16plus`) and ViT-7B (`vit7b16`) have no LT-DETR support at all yet, pretrained or not. That's not part of this bug, just flagging it.
### Root Cause / Where to Fix
Two places need changes, to match how DINOv2 already works:
1. **`src/lightly_train/_models/dinov3/dinov3_package.py`** (`DINOv3Package.MODEL_NAME_TO_INFO`): only `vitt16-notpretrained` and `vitt16plus-notpretrained` exist as "no weights" backbone entries (`functools.partial(backbones.dinov3_vit*, pretrained=False)`). We need to add the same kind of entry for `convnext-tiny/small/base/large`, `vits16`, `vitb16`, and `vitl16`.
2. **`src/lightly_train/_task_models/ltdetr_object_detection/config.py`**: needs one correctly-wired `*-notpretrained-ltdetr` config class per size, following the DINOv2 pattern (see `DINOv2ViTSmallNotPretrained`, `DINOv2ViTBaseNotPretrained`, `DINOv2ViTLargeNotPretrained`, `DINOv2ViTGiantNotPretrained`, around lines 1011-1098). Each class should point `backbone_name` at the new entry from step 1. The existing `dinov3/vitt16-notpretrained-ltdetr` also needs fixing — right now it points to the same class as the pretrained version instead of its own.
### Expected Behavior
Every DINOv3 LT-DETR backbone size should have a working `dinov3/-notpretrained-ltdetr` name that actually gives you a randomly-initialized backbone — the same way DINOv2 already works, with no extra `backbone_args` needed.
### Documentation
`docs/source/object_detection.md` lists all supported LT-DETR model names, but never mentions `notpretrained` for DINOv2 or DINOv3. So this option isn't discoverable without reading the source. Once the names are fixed/added, please also document them there.
### Minor Related Gap (DINOv2)
Found while comparing against DINOv2: `dinov2/vits14-noreg-ltdetr` (the legacy no-registers variant) has no `-notpretrained-ltdetr` version, even though the underlying `vits14-noreg-notpretrained` entry already exists and is unused. Easy to add alongside the DINOv3 fix.
### Additional Context
Found this while checking whether `pretrained: True` needs to be set explicitly to get a DINOv3-pretrained backbone in LT-DETR (it doesn't — that already works automatically and is unrelated to this bug). This issue is about the opposite case: reliably getting a backbone that is **not** pretrained.
If anything above is unclear, please leave a comment and I'll clarify.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read src/lightly_train/_models/dinov3/dinov3_package.py and src/lightly_train/_task_models/ltdetr_object_detection/config.py, using the DINOv2 not-pretrained classes as the wiring reference. Update the documented aliases in docs/source/object_detection.md, then verify every listed DINOv3 size and the DINOv2 no-registers variant resolves to a distinct no-weights configuration without extra backbone_args.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, documentation, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 66/100