intel / intel/llm-scaler

Determining which model has best compatability with llm-scaler

Open
#348 12 comments 0 reactions 1 assignee Claimed by @gc-fu View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

So I'm taking the plunge and giving a B70 a shot, and am preparing my system by switching from Debian to Ubuntu25.10+llm-scaler.

My question is, since llm-scaler isn't exactly the same as vLLM, how do I go about deciding what a better model to run on a single B70 is?

Or more specifically: Is there a place with rules like "Int4 works best since it runs at the hardware level"?

For example, HuggingFace has three potential Qwen3.5 35B models that should fit on a B70:

Two from Intel:
- [Intel/Qwen3.5-35B-A3B-gguf-q2ks-mixed-AutoRound](https://huggingface.co/Intel/Qwen3.5-35B-A3B-gguf-q2ks-mixed-AutoRound)
- [Intel/Qwen3.5-35B-A3B-int4-AutoRound](https://huggingface.co/Intel/Qwen3.5-35B-A3B-int4-AutoRound)

One from Qwen:
- [Qwen/Qwen3.5-35B-A3B-GPTQ-Int4](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-GPTQ-Int4)

How do I know which one to start with, and thus leading to which llm-scaler/vLLM flags to use for launch?

Thanks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.