Determining which model has best compatability with llm-scaler
- Dominant language
- C++
- Stars
- 529
- Forks
- 80
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 38
Description
So I'm taking the plunge and giving a B70 a shot, and am preparing my system by switching from Debian to Ubuntu25.10+llm-scaler.
My question is, since llm-scaler isn't exactly the same as vLLM, how do I go about deciding what a better model to run on a single B70 is?
Or more specifically: Is there a place with rules like "Int4 works best since it runs at the hardware level"?
For example, HuggingFace has three potential Qwen3.5 35B models that should fit on a B70:
Two from Intel:
- [Intel/Qwen3.5-35B-A3B-gguf-q2ks-mixed-AutoRound](https://huggingface.co/Intel/Qwen3.5-35B-A3B-gguf-q2ks-mixed-AutoRound)
- [Intel/Qwen3.5-35B-A3B-int4-AutoRound](https://huggingface.co/Intel/Qwen3.5-35B-A3B-int4-AutoRound)
One from Qwen:
- [Qwen/Qwen3.5-35B-A3B-GPTQ-Int4](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-GPTQ-Int4)
How do I know which one to start with, and thus leading to which llm-scaler/vLLM flags to use for launch?
Thanks.
Contributor guide
Assessment
This issue has not been assessed yet.