DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA3

Why only mlp2x_gelu pretrained weights when code supports multiple projector types?

Open
#94 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.2k
Forks
89
PR merge metrics
No merged PRs in 30d

Description

I notice that the VideoLLaMA3 codebase has support for multiple projector architectures:

mlp2x_gelu (currently used in all pretrained models)
mlp3x_gelu
linear
simp_spatial_conv
However, all the official released models (VideoLLaMA3-2B, VideoLLaMA3-7B, etc.) only use mlp2x_gelu projector.

Why were other projector types not trained and released as alternatives?
Are there plans to release variants with different projector types?
Can you share insights on the performance comparison between different projector types?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the projector architecture names listed in the issue: mlp2x_gelu, mlp3x_gelu, linear, and simp_spatial_conv. Done would require a maintainer-provided explanation of why released models use mlp2x_gelu, whether alternatives are planned, and how their performance compares; no files or tests are identified.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.