lightly-ai / lightly-ai/lightly-train

[BUG] <Pre-training weight loading issue>

Open
#435 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug dinov3 segmentation
Dominant language
Python
Stars
1.7k
Forks
116
Avg merge
2d 21h
Merged PRs (30d)
6

Description

I attempted to train my dataset using dinov3-eomt semantic segmentation. The code is as follows.

```
import lightly_train
if __name__ == "__main__":
lightly_train.train_semantic_segmentation(
out="/my_experiment",
model="dinov3/vitl16-eomt",
data={
},
"val": {
},
"classes": { # Classes in the dataset
},
# Optional, classes that are in the dataset but should be ignored during
# training.
"ignore_classes": [0],
},
devices=1,
batch_size=16,
)
```

However, when loading the local weight path, the following error occurred. I downloaded the weight file according to the code. How could there be an error? And, when I use the pre-trained weights of dinov2, this problem will not occur.

> Traceback (most recent call last):
> File "/home/user/.conda/envs/seg/lib/python3.11/site-packages/lightly_train/_configs/validate.py", line 30, in pydantic_model_validate
> return model.model_validate(obj)
> ^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/.conda/envs/seg/lib/python3.11/site-packages/pydantic/main.py", line 716, in model_validate
> return cls.__pydantic_validator__.validate_python(
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> pydantic_core._pydantic_core.ValidationError: 1 validation error for DINOv3EoMTSemanticSegmentationTrainArgs
> load_weights
> Extra inputs are not permitted [type=extra_forbidden, input_value=False, input_type=bool]
> For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/home/user/cyshi_lx/lightly-train-main/lightly-train-main/train.py", line 4, in
> lightly_train.train_semantic_segmentation(
> File "/home/user/.conda/envs/seg/lib/python3.11/site-packages/lightly_train/_commands/train_task.py", line 443, in train_semantic_segmentation
> return _train_task(config_cls=SemanticSegmentationTrainTaskConfig, **locals())
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/.conda/envs/seg/lib/python3.11/site-packages/lightly_train/_commands/train_task.py", line 475, in _train_task
> _train_task_from_config(config=config)
> File "/home/user/.conda/envs/seg/lib/python3.11/site-packages/lightly_train/_commands/train_task.py", line 656, in _train_task_from_config
> train_model = train_model_cls(
> ^^^^^^^^^^^^^^^^
> File "/home/user/.conda/envs/seg/lib/python3.11/site-packages/lightly_train/_task_models/dinov3_eomt_semantic_segmentation/train_model.py", line 190, in __init__
> self.model = DINOv3EoMTSemanticSegmentation(
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/.conda/envs/seg/lib/python3.11/site-packages/lightly_train/_task_models/dinov3_eomt_semantic_segmentation/task_model.py", line 133, in __init__
> backbone = DINOV3_PACKAGE.get_model(
> ^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/.conda/envs/seg/lib/python3.11/site-packages/lightly_train/_models/dinov3/dinov3_package.py", line 177, in get_model
> model = model_builder(**args)
> ^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/.conda/envs/seg/lib/python3.11/site-packages/lightly_train/_models/dinov3/dinov3_src/hub/backbones.py", line 361, in dinov3_vitl16
> return _make_dinov3_vit(
> ^^^^^^^^^^^^^^^^^
> File "/home/user/.conda/envs/seg/lib/python3.11/site-packages/lightly_train/_models/dinov3/dinov3_src/hub/backbones.py", line 154, in _make_dinov3_vit
> model.load_state_dict(state_dict, strict=True)
> File "/home/user/.conda/envs/seg/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2581, in load_state_dict
> raise RuntimeError(
> RuntimeError: Error(s) in loading state_dict for DinoVisionTransformer:
> Unexpected key(s) in state_dict: "local_cls_norm.weight", "local_cls_norm.bias".

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the reported dinov3/vitl16-eomt model and local weights, then inspect _models/dinov3/dinov3_src/hub/backbones.py and _models/dinov3/dinov3_package.py around model construction and state-dict loading. Compare the checkpoint keys with the DinoVisionTransformer definition and confirm that compatible pretrained weights load without the reported unexpected keys.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.