[Bug] Ktransformers error message when using my Tesla P40 GPU

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

### Describe the bug

2025-06-17 16:47:42,686 DEBUG /disk/ktransformers/ktransformers/server/backend/context_manager.py[23]: Creating Context Manager
2025-06-17 16:47:42,687 INFO /disk/ktransformers/ktransformers/server/main.py[23]: Creating SQL tables
2025-06-17 16:47:42,689 INFO /disk/ktransformers/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
INFO: Started server process [2616]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:10002 (Press CTRL+C to quit)
INFO: 192.168.0.125:46378 - "GET /v1/models HTTP/1.1" 200 OK
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
/disk/ktransformers/env/lib/python3.11/site-packages/pydantic/main.py:519: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected `list[dict[str, any]]` - serialized value may not be as expected [input_value={}, input_type=dict])
return self.__pydantic_serializer__.to_json(
INFO: 192.168.0.125:46946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-06-17 17:15:58,677 WARNING /disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py[178]: change system to user
2025-06-17 17:15:58,677 WARNING /disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py[184]: merge two adjacent user messages
2025-06-17 17:15:58,724 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py[209]: get input ids of shape torch.Size([1, 575])
2025-06-17 17:15:58,724 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[144]: input_ids: torch.Size([1, 575])
2025-06-17 17:15:58,725 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[169]: same prefix len: 0
2025-06-17 17:15:58,736 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[178]: input_ids: torch.Size([1, 575])
2025-06-17 17:15:58,736 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[179]: generate_ids: torch.Size([1, 0])
2025-06-17 17:15:58,794 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[194]: cache position: 0 to 575
ERROR: Exception in ASGI application
+ Exception Group Traceback (most recent call last):
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 263, in __call__
| async with anyio.create_task_group() as task_group:
| File "/disk/ktransformers/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/disk/ktransformers/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
| await super().__call__(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
| raise exc
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
| await self.app(scope, receive, _send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
| await self.app(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
| await route.handle(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
| await response(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__
| with collapse_excgroups():
| File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
| self.gen.throw(typ, value, traceback)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
| await func()
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
| async for chunk in self.body_iterator:
| File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
| async for event in async_events:
| File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
| async for event in async_events:
| File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
| async for event in async_events:
| File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
| async for event in async_events:
| File "/disk/ktransformers/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
| async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
| File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
| async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
| File "/disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
| for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
| response = gen.send(None)
| ^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
| logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
| logits = self.model(
| ^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
| outputs = self.model(
| ^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/operators/models.py", line 751, in forward
| layer_outputs = decoder_layer(
| ^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1207, in forward
| hidden_states, self_attn_weights, present_key_value = self.self_attn(
| ^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'
+------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/disk/ktransformers/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
await self.middleware_stack(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__
with collapse_excgroups():
File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
await func()
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
async for chunk in self.body_iterator:
File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
async for event in async_events:
File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
async for event in async_events:
File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
async for event in async_events:
File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
async for event in async_events:
File "/disk/ktransformers/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
File "/disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
response = gen.send(None)
^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
logits = self.model(
^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
outputs = self.model(
^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/operators/models.py", line 751, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1207, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'
INFO: 192.168.0.125:46962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-06-17 17:15:59,655 WARNING /disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py[178]: change system to user
2025-06-17 17:15:59,655 WARNING /disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py[184]: merge two adjacent user messages
2025-06-17 17:15:59,657 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py[209]: get input ids of shape torch.Size([1, 575])
2025-06-17 17:15:59,657 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[144]: input_ids: torch.Size([1, 575])
2025-06-17 17:15:59,800 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[169]: same prefix len: 574
2025-06-17 17:15:59,801 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[178]: input_ids: torch.Size([1, 1])
2025-06-17 17:15:59,801 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[179]: generate_ids: torch.Size([1, 574])
2025-06-17 17:15:59,801 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[194]: cache position: 574 to 575
ERROR: Exception in ASGI application
+ Exception Group Traceback (most recent call last):
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 263, in __call__
| async with anyio.create_task_group() as task_group:
| File "/disk/ktransformers/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/disk/ktransformers/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
| await super().__call__(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
| raise exc
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
| await self.app(scope, receive, _send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
| await self.app(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
| await route.handle(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
| await response(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__
| with collapse_excgroups():
| File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
| self.gen.throw(typ, value, traceback)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
| await func()
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
| async for chunk in self.body_iterator:
| File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
| async for event in async_events:
| File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
| async for event in async_events:
| File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
| async for event in async_events:
| File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
| async for event in async_events:
| File "/disk/ktransformers/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
| async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
| File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
| async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
| File "/disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
| for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
| response = gen.send(None)
| ^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
| logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
| logits = self.model(
| ^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
| outputs = self.model(
| ^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/operators/models.py", line 751, in forward
| layer_outputs = decoder_layer(
| ^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1207, in forward
| hidden_states, self_attn_weights, present_key_value = self.self_attn(
| ^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'
+------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/disk/ktransformers/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
await self.middleware_stack(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__
with collapse_excgroups():
File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
await func()
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
async for chunk in self.body_iterator:
File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
async for event in async_events:
File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
async for event in async_events:
File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
async for event in async_events:
File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
async for event in async_events:
File "/disk/ktransformers/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
File "/disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
response = gen.send(None)
^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
logits = self.model(
^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
outputs = self.model(
^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/operators/models.py", line 751, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1207, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'
INFO: 192.168.0.125:46964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-06-17 17:16:00,358 WARNING /disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py[178]: change system to user
2025-06-17 17:16:00,359 WARNING /disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py[184]: merge two adjacent user messages
2025-06-17 17:16:00,361 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py[209]: get input ids of shape torch.Size([1, 575])
2025-06-17 17:16:00,361 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[144]: input_ids: torch.Size([1, 575])
2025-06-17 17:16:00,379 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[169]: same prefix len: 574
2025-06-17 17:16:00,380 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[178]: input_ids: torch.Size([1, 1])
2025-06-17 17:16:00,380 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[179]: generate_ids: torch.Size([1, 574])
2025-06-17 17:16:00,380 DEBUG /disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[194]: cache position: 574 to 575
ERROR: Exception in ASGI application
+ Exception Group Traceback (most recent call last):
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 263, in __call__
| async with anyio.create_task_group() as task_group:
| File "/disk/ktransformers/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/disk/ktransformers/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
| await super().__call__(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
| raise exc
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
| await self.app(scope, receive, _send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
| await self.app(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
| await route.handle(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
| await response(scope, receive, send)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__
| with collapse_excgroups():
| File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
| self.gen.throw(typ, value, traceback)
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
| await func()
| File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
| async for chunk in self.body_iterator:
| File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
| async for event in async_events:
| File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
| async for event in async_events:
| File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
| async for event in async_events:
| File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
| async for event in async_events:
| File "/disk/ktransformers/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
| async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
| File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
| async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
| File "/disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
| for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
| response = gen.send(None)
| ^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
| logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
| logits = self.model(
| ^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
| outputs = self.model(
| ^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/operators/models.py", line 751, in forward
| layer_outputs = decoder_layer(
| ^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1207, in forward
| hidden_states, self_attn_weights, present_key_value = self.self_attn(
| ^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'
+------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/disk/ktransformers/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
await self.middleware_stack(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__
with collapse_excgroups():
File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
await func()
File "/disk/ktransformers/env/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
async for chunk in self.body_iterator:
File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
async for event in async_events:
File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
async for event in async_events:
File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
async for event in async_events:
File "/disk/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
async for event in async_events:
File "/disk/ktransformers/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
File "/disk/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
response = gen.send(None)
^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
logits = self.model(
^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
outputs = self.model(
^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/operators/models.py", line 751, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1207, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk/ktransformers/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'

### Reproduction

python ktransformers/server/main.py --model_path /disk/models/DeepSeek-R1-GGUF --gguf_path /disk/models/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S --cpu_infer 62 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml --port 10002 --chunk_size 256 --max_new_tokens 1024 --max_batch_size 4 --port 10002 --cache_lens 32768 --backend_type ktransformers

### Environment

Debian 12, Tesla P40 Gpu

pip list
Package Version Editable project location
------------------------ ---------------------- -------------------------------------------------
accelerate 1.7.0
annotated-types 0.7.0
anyio 4.9.0
blessed 1.21.0
blobfile 3.0.0
build 1.2.2.post1
certifi 2025.4.26
charset-normalizer 3.4.2
click 8.1.7
colorlog 6.9.0
distro 1.9.0
fastapi 0.115.12
filelock 3.13.1
fire 0.7.0
flashinfer-python 0.2.3 /disk/ktransformers/third_party/custom_flashinfer
fsspec 2024.6.1
greenlet 3.2.3
h11 0.16.0
hf-xet 1.1.3
httpcore 1.0.9
httpx 0.28.1
huggingface-hub 0.33.0
idna 3.10
Jinja2 3.1.3
jiter 0.10.0
jsonpatch 1.33
jsonpointer 3.0.0
ktransformers 0.3.1+cu128torch25avx2 /disk/ktransformers
langchain 0.3.25
langchain-core 0.3.65
langchain-text-splitters 0.3.8
langsmith 0.3.45
lxml 5.4.0
MarkupSafe 2.1.5
mpmath 1.3.0
networkx 3.3
ninja 1.11.1.4
numpy 2.1.2
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
openai 1.86.0
orjson 3.10.18
packaging 24.1
pillow 11.0.0
pip 23.0.1
protobuf 6.31.1
psutil 7.0.0
pycryptodomex 3.23.0
pydantic 2.11.6
pydantic_core 2.33.2
pyproject_hooks 1.2.0
PyYAML 6.0.2
regex 2024.11.6
requests 2.32.4
requests-toolbelt 1.0.0
safetensors 0.5.3
sentencepiece 0.2.0
setuptools 66.1.1
sniffio 1.3.1
SQLAlchemy 2.0.41
starlette 0.46.2
sympy 1.13.1
tenacity 9.1.2
termcolor 3.1.0
tiktoken 0.9.0
tokenizers 0.21.1
torch 2.5.1+cu124
torchaudio 2.5.1+cu124
torchvision 0.20.1+cu124
tqdm 4.67.1
transformers 4.51.3
transient-package 1.0.1.post3
triton 3.1.0
triton-pascal 3.3.0
typing_extensions 4.14.0
typing-inspection 0.4.1
urllib3 2.4.0
uvicorn 0.34.3
wcwidth 0.2.13
wheel 0.44.0
zstandard 0.23.0

Start with ktransformers/server/backend/interfaces/ktransformers.py around prefill and ktransformers/models/modeling_deepseek_v3.py around the model forward path shown in the traceback. Inspect how flashinfer_attn.forward is called and reproduce the request through the server entry point. Done means the reported chat-completions request no longer raises the unexpected attention_mask TypeError on the Tesla P40 setup.

Written by the indexing model from the issue text.