intel / intel/llm-scaler

`enable-expert-parallel` crashes with `VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=0` and `sym_int4` autoquant

Open
#438 7 comments 0 reactions 1 assignee Claimed by @Wesley-Du View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

Early playing around with the new `intel/llm-scaler-vllm:0.14.0-b8.3` release is very promising for Qwen 3.6 on dual B60s, but I've had to take out `--enable-expert-parallel` from my configs. In the b8.1 release, it was giving me a significant load speedup for Qwen3.6 35B A3B models. I have not tested with b8.2 releases.

I'm attaching the vllm log, my `compose.yaml` (renamed with .txt so github will accept it) as well as the .env file referenced in the compose (again with altered file extension). The non-default vllm launch args are in the logfile.

[enable-expert-parallel.error.log](https://github.com/user-attachments/files/28375155/enable-expert-parallel.error.log)

[compose.yaml.txt](https://github.com/user-attachments/files/28375203/compose.yaml.txt)

[lsv-container.env.txt](https://github.com/user-attachments/files/28375191/lsv-container.env.txt)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.