PaddlePaddle / PaddlePaddle/FastDeploy

单机8块P800运行Qwen3-235B-A22B总是loadlayers开始的时候崩溃, xpu_ipc_get_memhandle failed 。

Open
#8,102 0 comments 0 reactions 1 assignee View on GitHub

@gongshaotian is already working on this.

Since Jul 16, 2026.

Dominant language
Python
Stars
3.7k
Forks
756
Avg merge
19h 28m
Merged PRs (30d)
4

Description

INFO 2026-07-16 14:43:22,869 1748 engine.py[line:151] Waiting for worker processes to be ready...
Loading Weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [01:43<00:00, 1.04s/it]
Loading Layers: 0%| | 0/100 [00:11<?, ?it/s]
ERROR 2026-07-16 14:45:22,612 1748 engine.py[line:160] Failed to launch worker processes, check log/workerlog.* for more details.

ERROR 2026-07-16 14:45:31,391 1748 engine.py[line:447] Error extracting sub services: [Errno 3] No such process, Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/engine/engine.py", line 444, in _exit_sub_services
pgid = os.getpgid(self.worker_proc.pid) Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/engine/../worker/worker_process.py", line 1296, in
run_worker_proc()
File "/usr/local/lib/python3.10/dist-packages/paddle/base/dygraph/base.py", line 406, in _decorate_function
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/engine/../worker/worker_process.py", line 1281, in run_worker_proc
worker_proc.initialize_kv_cache()
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/engine/../worker/worker_process.py", line 659, in initialize_kv_cache
available_kv_cache_memory = self.worker.determine_available_memory()
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/worker/xpu_worker.py", line 130, in determine_available_memory
self.model_runner.profile_run()
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/worker/xpu_model_runner.py", line 1711, in profile_run
self.initialize_kv_cache(profile=True)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/worker/xpu_model_runner.py", line 1270, in initialize_kv_cache
set_data_ipc(key_cache, key_cache_name)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/model_executor/ops/xpu/fastdeploy_ops/init.py", line 2000, in set_data_ipc
outs = _C_ops._run_custom_op("set_data_ipc", tmp_input,shm_name)
OSError: xpu_ipc_get_memhandle failed

使用的是fastdeploy 2.5镜像,启动参数及模型启动参数均参考了文档的参数。请问有没有哪位碰到过类似问题。

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.