OpenEuroLLM / OpenEuroLLM/Taskboard
Evaluate 32B checkpoints on both first and conservative run
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 3
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Goal
We need to evaluate 32B checkpoints on both runs to make a data-driven comparison, currently we have just a few evaluations with not much multilingual and they do not consider annealing simulation.
- convert 32B checkpoints and push to HF @maxidl
- generate simulated annealed checkpoints with Nemotron tool @maxidl
- make sure current workflow with vLLM works on all tasks
- evaluate all checkpoints and simulated annealing for DCLM Core
- evaluate all checkpoints and simulated annealing for multilingual
For WSM, perhaps here we just consider the annealing on last 300B tokens to stay close to results reported on WSM wdyt @maxidl ? I also do not mind picking anything from cosine / sqrt / linear but I have found Merge Cos. to work best on prelude.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the current vLLM workflow and the Nemotron tool used to generate simulated annealed checkpoints, then compare the evaluation approach with issue 406. Done means converting the 32B checkpoints, generating the requested simulations, confirming all tasks run, and completing DCLM Core and multilingual evaluations, with the WSM annealing scope decided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100