`enable-expert-parallel` crashes with `VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=0` and `sym_int4` autoquant
- Dominant language
- C++
- Stars
- 529
- Forks
- 80
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 38
Description
Early playing around with the new `intel/llm-scaler-vllm:0.14.0-b8.3` release is very promising for Qwen 3.6 on dual B60s, but I've had to take out `--enable-expert-parallel` from my configs. In the b8.1 release, it was giving me a significant load speedup for Qwen3.6 35B A3B models. I have not tested with b8.2 releases.
I'm attaching the vllm log, my `compose.yaml` (renamed with .txt so github will accept it) as well as the .env file referenced in the compose (again with altered file extension). The non-default vllm launch args are in the logfile.
[enable-expert-parallel.error.log](https://github.com/user-attachments/files/28375155/enable-expert-parallel.error.log)
[compose.yaml.txt](https://github.com/user-attachments/files/28375203/compose.yaml.txt)
[lsv-container.env.txt](https://github.com/user-attachments/files/28375191/lsv-container.env.txt)
Contributor guide
Assessment
This issue has not been assessed yet.