intel / intel/llm-scaler

Issues noticed when trying https://github.com/intel/llm-scaler/releases/tag/vllm-0.14.0-b8.3

Open
#439 1 comment 0 reactions 1 assignee Claimed by @gc-fu View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

I noticed when I tested the new 0.14.0-b8.3 on Qwen/Qwen3.6-35b-a3b moe

I used llm scaler vllm docker image 0.14.0-b8.3 on ubuntu with 2x b70 with tags like
--quantization fp8
--max-model-len 256000
--gpu-memory-utilization 0.92

--dtype=bfloat16
The custom SYCL/ESIMD MoE kernel (/tmp/custom-esimd-kernels-vllm/csrc/moe_batch/moe.sycl) only supports float16, not bfloat16.

--kv_cache_dtype fp8
Starts serving but crashes. The page_attn_decode kernel in eagle.sycl:341 seems to rejects the fp8 KV cache.

--default-chat-template-kwargs '{"enable_thinking": true}'
--reasoning-parser qwen3
Is ignored and the model is thinking out loud, both when added or removed with true/false statement.
Its start output with: Here's a thinking process: and ends reasoning content with something like: Proceed. Output matches the refined response.✅
I have tried passing through --trust-remote-code and/or --chat-template /llm/models/Qwen3.6-35B-A3B/chat_template.jinja and I still cant get it to separate reasoning content from content.
Looks like the output might not be generating < think > < /think > correctly. The problem seems to be that the reasoning parser plugin isn't loaded in this V1 engine build (reasoning_parser_plugin=''), so reasoning_content stays None.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.