Quantitative testing of models on HF
- Dominant language
- C++
- Stars
- 229
- Forks
- 38
- PR merge metrics
- No merged PRs in 30d
Description
@ngxson suggested to do a larger sweep of compatibility tests on [this llama.cpp PR](https://github.com/ggerganov/llama.cpp/pull/9639#issuecomment-2510269206) (cc/ @Vaibhavs10 & @Rocketknight1)
To start with, I've added the models listed in https://aiworld.eu/embed/model/model/treemap (top liked / downloaded) to the tested [MODEL_IDS](https://github.com/google/minja/blob/main/tests/CMakeLists.txt#L22).
Known failures:
- [fireworks-ai/llama-3-firefunction-v2](https://huggingface.co/fireworks-ai/llama-3-firefunction-v2) -> https://github.com/google/minja/issues/7
- [ai21labs/AI21-Jamba-1.5-Large](https://huggingface.co/ai21labs/AI21-Jamba-1.5-Large) -> https://github.com/google/minja/issues/8
Some others are playing hard to get (where are the templates for ~~[ai21labs/Jamba-v0.1](https://huggingface.co/ai21labs/Jamba-v0.1)~~, [apple/OpenELM-1_1B-Instruct ](https://huggingface.co/apple/OpenELM-1_1B-Instruct ), [dreamgen/WizardLM-2-7B](https://huggingface.co/dreamgen/WizardLM-2-7B), [xai-org/grok-1](https://huggingface.co/xai-org/grok-1)?), but otherwise so far so good :-)
Contributor guide
Assessment
This issue has not been assessed yet.