OpenEuroLLM / OpenEuroLLM/Taskboard
MoE hyperparameter scaling law experiments throughput testing on JUPITER
Open
@dharmsen is already working on this.
Since Jun 2, 2026.
T4.1 - Optimization HPC
T4.2 - model exploration
- Dominant language
- No language data
- Stars
- 3
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
While preparing for the MoE hyperparameter scaling law experiments on LEONARDO, I ran throughput tests on JUPITER as a parallel baseline and sanity check. The configs can be found here.
These are the results I have obtained so far:
| model | nodes | mbs | gbs | tflops/s/gpu | tokens/s/gpu |
|---|---|---|---|---|---|
| 0.1B | 1 | 1 | 64 | 24 | 24644 |
| 0.1B | 1 | 2 | 64 | 43 | 44488 |
| 0.1B | 1 | 4 | 64 | 76 | 77509 |
| 0.1B | 1 | 8 | 64 | 114 | 116416 |
| 0.2B | 1 | 1 | 64 | 16 | 8720 |
| 0.2B | 1 | 2 | 64 | 57 | 29996 |
| 0.2B | 1 | 4 | 64 | 106 | 56143 |
| 0.2B | 1 | 8 | 64 | 146 | 76961 |
| 0.4B | 1 | 1 | 64 | 40 | 9295 |
| 0.4B | 1 | 2 | 64 | 80 | 18367 |
| 0.4B | 1 | 4 | 64 | 145 | 33391 |
| 0.4B | 1 | 8 | 64 | 162 | 37246 |
I am currently facing multi-node NCCL issues, so results for 0.9B and possible throughput scaling to high node counts is still not available. Will continue work on this soon, especially if we are considering running these experiments on JUPITER.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.