google / google/minja

Quantitative testing of models on HF

Open
#6 3 comments 3 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
229
Forks
38
PR merge metrics
No merged PRs in 30d

Description

@ngxson suggested to do a larger sweep of compatibility tests on [this llama.cpp PR](https://github.com/ggerganov/llama.cpp/pull/9639#issuecomment-2510269206) (cc/ @Vaibhavs10 & @Rocketknight1)

To start with, I've added the models listed in https://aiworld.eu/embed/model/model/treemap (top liked / downloaded) to the tested [MODEL_IDS](https://github.com/google/minja/blob/main/tests/CMakeLists.txt#L22).

Known failures:
- [fireworks-ai/llama-3-firefunction-v2](https://huggingface.co/fireworks-ai/llama-3-firefunction-v2) -> https://github.com/google/minja/issues/7
- [ai21labs/AI21-Jamba-1.5-Large](https://huggingface.co/ai21labs/AI21-Jamba-1.5-Large) -> https://github.com/google/minja/issues/8

Some others are playing hard to get (where are the templates for ~~[ai21labs/Jamba-v0.1](https://huggingface.co/ai21labs/Jamba-v0.1)~~, [apple/OpenELM-1_1B-Instruct ](https://huggingface.co/apple/OpenELM-1_1B-Instruct ), [dreamgen/WizardLM-2-7B](https://huggingface.co/dreamgen/WizardLM-2-7B), [xai-org/grok-1](https://huggingface.co/xai-org/grok-1)?), but otherwise so far so good :-)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.