OpenEuroLLM / OpenEuroLLM/Taskboard
Hybrid Model Tests
Open
@kpoeppel is already working on this.
Since Aug 4, 2026.
T4.4 - model training
- Dominant language
- No language data
- Stars
- 3
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Goal
Testing Hybrid model architectures for scaling in comparison to FullAttn
Description
Test Mamba2, GDN, SWA, mLSTM hybrids compared to FullAttn on multilingual scaling best (bs, lr) points
Deliverable scope
Provide a comparison on different scales showing if we should switch to a hybrid arch.
Preliminary data:
At the moment the speed is: SWA > mLSTM, FullAttn, > Mamba2 > GDN. In terms of GPU-hrs based best performance Mamba2 still seems to work best.
Dependencies
none
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.