[Bug] Deepseek R1 Quantitative Version Model Dialogue Error

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

### Describe the bug

error occurred during the conversation with Deepseek R1 0528 quantized：

025-06-16 03:56:35,045 DEBUG /home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[209]: get input ids of shape torch.Size([1, 9])
2025-06-16 03:56:35,046 DEBUG /home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[144]: input_ids: torch.Size([1, 9])
2025-06-16 03:56:35,046 DEBUG /home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[169]: same prefix len: 0
2025-06-16 03:56:35,049 DEBUG /home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[178]: input_ids: torch.Size([1, 9])
2025-06-16 03:56:35,049 DEBUG /home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[179]: generate_ids: torch.Size([1, 0])
2025-06-16 03:56:35,051 DEBUG /home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[194]: cache position: 0 to 9
ERROR: Exception in ASGI application
+ Exception Group Traceback (most recent call last):
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 263, in __call__
| async with anyio.create_task_group() as task_group:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
| await super().__call__(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
| raise exc
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
| await self.app(scope, receive, _send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
| await self.app(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
| await route.handle(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
| await response(scope, receive, send)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__
| with collapse_excgroups():
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/contextlib.py", line 158, in __exit__
| self.gen.throw(typ, value, traceback)
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
| await func()
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
| async for chunk in self.body_iterator:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
| async for event in async_events:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
| async for event in async_events:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
| async for event in async_events:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
| async for event in async_events:
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
| async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
| async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
| for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
| response = gen.send(None)
| ^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
| logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
| logits = self.model(
| ^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
| outputs = self.model(
| ^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/models.py", line 751, in forward
| layer_outputs = decoder_layer(
| ^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1207, in forward
| hidden_states, self_attn_weights, present_key_value = self.self_attn(
| ^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 723, in forward
| return self.forward_linux_flashinfer(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 508, in forward_linux_flashinfer
| attn_output = flash_attn_func(
| ^^^^^^^^^^^^^^^
| NameError: name 'flash_attn_func' is not defined
+------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__
with collapse_excgroups():
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
await func()
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
async for chunk in self.body_iterator:
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
async for event in async_events:
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
async for event in async_events:
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
async for event in async_events:
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
async for event in async_events:
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
logits = self.model(
^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
outputs = self.model(
^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/models.py", line 751, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1207, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 723, in forward
return self.forward_linux_flashinfer(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dbcszh/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 508, in forward_linux_flashinfer
attn_output = flash_attn_func(
^^^^^^^^^^^^^^^
NameError: name 'flash_attn_func' is not defined

### Reproduction

python ktransformers/server/main.py --model_path deepseek-ai/DeepSeek-R1-0528 --gguf_path ../llama.cpp/models_hf/models/DeepSeek-R1-0528-GGUF/UD-Q2_K_XL --model_name DeepSeek-R1-Q2:671B --max_new_tokens 16384 --cpu_infer 48 --temperature 0.6 --port 10001

### Environment

ubuntu 22.04
cuda 12.8
torch 2.7.1+cu126
flashinfer-python 0.2.3
ktransformers 0.3.1+cu124torch27avx2

Start by reproducing the DeepSeek R1 0528 quantized conversation and inspect the traceback through server/backend/interfaces/ktransformers.py, models/modeling_deepseek_v3.py, and operators/models.py. Trace the failure from prefill and chunk_prefill into self-attention, then verify that the conversation completes without the reported ASGI exception.

Written by the indexing model from the issue text.