[Bug] DeepSeek-R1-Q8_0使用fp8推理时报错

检查清单

• 1. 我已经搜索过相关问题，但未能获得预期的帮助
• 2. 该问题在最新版本中尚未修复
• 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
• 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
• 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

参考教程https://kvcache-ai.github.io/ktransformers/en/fp8_kernel.html

我在4090D上分别下载了贵方提供的DS-R1-q4km-fp8权重，推理过程正常

也使用了合并脚本，merge_tensors/merge_safetensor_gguf.py自行合并了DS-V3-0324-q4km和DS-V3-0324官方safetensors权重，并运行，推理过程正常

但是当我使用合并脚本，合并DS-R1-Q8_0权重和DS-R1官方权重时，合并没有问题，得到safetensors，但是推理过程报错

2025-04-01 14:37:06,843 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
2025-04-01 14:37:07,242 INFO /home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
2025-04-01 14:37:07,244 INFO /home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
2025-04-01 14:37:47,689 DEBUG /home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/backend/context_manager.py[21]: Creating Context Manager
INFO: Started server process [24115]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:20000 (Press CTRL+C to quit)
2025-04-01 14:38:07,492 DEBUG /home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[197]: get input ids of shape torch.Size([1, 5])
2025-04-01 14:38:07,496 DEBUG /home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[133]: input_ids: torch.Size([1, 7])
2025-04-01 14:38:07,497 DEBUG /home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[159]: same prefix len: 0
2025-04-01 14:38:07,498 DEBUG /home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[168]: input_ids: torch.Size([1, 7])
2025-04-01 14:38:07,498 DEBUG /home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[170]: generate_ids: torch.Size([1, 0])
2025-04-01 14:38:07,498 DEBUG /home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[181]: cache position: 0 to 7
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/responses.py", line 268, in call
await wrap(partial(self.listen_for_disconnect, receive))
File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
await func()
File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/responses.py", line 233, in listen_for_disconnect
message = await receive()
^^^^^^^^^^^^^^^
File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 531, in receive
await self.message_event.wait()
File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/asyncio/locks.py", line 213, in wait
await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7e7117fbba50

During handling of the above exception, another exception occurred:

• Exception Group Traceback (most recent call last):
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
  | result = await app( # type: ignore[func-returns-value]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
  | return await self.app(scope, receive, send)
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
  | await super().call(scope, receive, send)
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/applications.py", line 112, in call
  | await self.middleware_stack(scope, receive, send)
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in call
  | raise exc
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in call
  | await self.app(scope, receive, _send)
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call
  | await self.app(scope, receive, send)
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in call
  | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  | raise exc
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  | await app(scope, receive, sender)
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/routing.py", line 715, in call
  | await self.middleware_stack(scope, receive, send)
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
  | await route.handle(scope, receive, send)
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
  | await self.app(scope, receive, send)
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
  | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  | raise exc
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  | await app(scope, receive, sender)
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
  | await response(scope, receive, send)
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/responses.py", line 261, in call
  | async with anyio.create_task_group() as task_group:
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 767, in aexit
  | raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
  | Traceback (most recent call last):
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
  | await func()
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/starlette/responses.py", line 245, in stream_response
  | async for chunk in self.body_iterator:
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
  | async for event in async_events:
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
  | async for event in async_events:
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
  | async for event in async_events:
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
  | async for event in async_events:
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 34, in inner
  | async for token in interface.inference(input_message,id):
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 217, in inference
  | async for v in super().inference(local_messages, thread_id):
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 389, in inference
  | for t in self.prefill(input_ids, self.check_is_new(thread_id)):
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
  | response = gen.send(None)
  | ^^^^^^^^^^^^^^
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in prefill
  | next_token = self.logits_to_token(logits[0, -1, :])
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/test/miniconda3/envs/ktransformers-env/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 232, in logits_to_token
  | last = torch.multinomial(probs, num_samples=1)
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | RuntimeError: CUDA error: device-side assert triggered
  | CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
  | For debugging consider passing CUDA_LAUNCH_BLOCKING=1
  | Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
  |
  +------------------------------------
  /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed.`

复现步骤

ktransformers --model_path /home/USERNAME/model/DeepSeek-R1 --gguf_path /mnt/modelfp8/DeepSeek-R1-q8-fp8 --cpu_infer 40 --max_new_tokens 8192 --cache_lens 32768 --total_context 32768 --temperature 0.6 --port 20000 --host 0.0.0.0 --force_think --api_key sk-xxxx --optimize_config_path /home/USERNAME/tools/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-fp8-linear-ggml-experts.yaml

环境信息

配置信息
内存：768G DDR4
GPU: 4090D
CPU: INTEL Sliver 4310 *2
未使用NUMA