[DOC] Can we use a bigger Dinov3 model?

Nobody has claimed this yet.

### ✏️ Suggested Improvement

I have located the key information. In task_model.py, the configuration mapping config_mapping for LtDetR explicitly lists the supported backbones:

config_mapping = {
"vitt16": ..., # ViT-Tiny
"vitt16plus": ..., # ViT-Tiny+
"vits16": ..., # ViT-Small
"vitb16": ..., # ViT-Base
"vitl16": ..., # ViT-Large
"convnext-tiny": ..., # ConvNeXt-Tiny
"convnext-small": ..., # ConvNeXt-Small
"convnext-base": ..., # ConvNeXt-Base
"convnext-large": ..., # ConvNeXt-Large
}
Currently, LtDetR does not support vit7b16

We need to do the following two things:

Register vit7b16 / vith16 in config_mapping.
Define the corresponding configuration classes (e.g., _HybridEncoderViT7B16Config, _RTDETRTransformerV2ViT7B16Config, etc.).

# after _HybridEncoderViTBConfig / _HybridEncoderViTLConfig

class _HybridEncoderViT7B16Config(_HybridEncoderConfig):
in_channels: list[int] = [4096, 4096, 4096] # 7B16 embed_dim=4096
feat_strides: list[int] = [8, 16, 32]
hidden_dim: int = 2048 # 2048
use_encoder_idx: list[int] = [2]
num_encoder_layers: int = 1
nhead: int = 16
dim_feedforward: int = 8192
dropout: float = 0.0
enc_act: str = "gelu"
expansion: float = 1.0
depth_mult: float = 1.0
act: str = "silu"

class _HybridEncoderViTH16Config(_HybridEncoderConfig):
in_channels: list[int] = [1280, 1280, 1280] # embed_dim=1280
feat_strides: list[int] = [8, 16, 32]
hidden_dim: int = 1024 # 1024
use_encoder_idx: list[int] = [2]
num_encoder_layers: int = 1
nhead: int = 16
dim_feedforward: int = 4096
dropout: float = 0.0
enc_act: str = "gelu"
expansion: float = 1.0
depth_mult: float = 1.0
act: str = "silu"
#decoder
class _RTDETRTransformerv2ViT7B16Config(_RTDETRTransformerv2Config):
feat_channels: list[int] = [2048, 2048, 2048]
hidden_dim: int = 2048
num_layers: int = 6
num_points: list[int] = [3, 6, 3]
dim_feedforward: int = 8192

class _RTDETRTransformerv2ViTH16Config(_RTDETRTransformerv2Config):
feat_channels: list[int] = [1024, 1024, 1024] # encoder hidden_dim=1024
hidden_dim: int = 1024
num_layers: int = 6
num_points: list[int] = [3, 6, 3]
dim_feedforward: int = 4096
# backbone wrapper
class _RTDETRBackboneWrapperViT7B16Config(PydanticConfig):
interaction_indexes: list[int] = [10, 20, 30]
finetune: bool = True
conv_inplane: int = 128
hidden_dim: int = 2048

class _RTDETRBackboneWrapperViTH16Config(PydanticConfig):
interaction_indexes: list[int] = [8, 16, 24]
finetune: bool = True
conv_inplane: int = 64
hidden_dim: int = 1024
#config_mapping
config_mapping = {
"vitt16": ...,
...
"vit7b16": (
_DINOv3LTDETRObjectDetectionViT7B16Config,
DINOv3STAs,
),
"vith16": (
_DINOv3LTDETRObjectDetectionViTH16Config,
DINOv3STAs,
),
# ...
}
# config
class _DINOv3LTDETRObjectDetectionViT7B16Config(_DINOv3LTDETRObjectDetectionConfig):
hybrid_encoder: _HybridEncoderViT7B16Config = Field(
default_factory=_HybridEncoderViT7B16Config
)
rtdetr_transformer: _RTDETRTransformerv2ViT7B16Config = Field(
default_factory=_RTDETRTransformerv2ViT7B16Config
)
rtdetr_postprocessor: _RTDETRPostProcessorConfig = Field(
default_factory=_RTDETRPostProcessorConfig
)
backbone_wrapper: _RTDETRBackboneWrapperViT7B16Config = Field(
default_factory=_RTDETRBackboneWrapperViT7B16Config
)

class _DINOv3LTDETRObjectDetectionViTH16Config(_DINOv3LTDETRObjectDetectionConfig):
hybrid_encoder: _HybridEncoderViTH16Config = Field(
default_factory=_HybridEncoderViTH16Config
)
rtdetr_transformer: _RTDETRTransformerv2ViTH16Config = Field(
default_factory=_RTDETRTransformerv2ViTH16Config
)
rtdetr_postprocessor: _RTDETRPostProcessorConfig = Field(
default_factory=_RTDETRPostProcessorConfig
)
backbone_wrapper: _RTDETRBackboneWrapperViTH16Config = Field(
default_factory=_RTDETRBackboneWrapperViTH16Config
)

Start in task_model.py by reading config_mapping and the existing ViT-B and ViT-L configuration classes. Add the corresponding ViT-7B/16 and ViT-H/16 configuration classes and mappings described in the issue. Done means LtDetR recognizes both backbones with their encoder, transformer, postprocessor, and backbone-wrapper configurations.

Written by the indexing model from the issue text.