ERROR - engine.py:781 - Task <EngineMainLoop> failed

I get an error when I did an inference with InternVL3-78B-AWQ model. How to solve it ?

lmdeploy==0.7.3

`model_path = "/home/ai-admin/llm-models/InternVL3-78B-AWQ"
image_path = "./downloads"

prompts = []
for file in os.listdir(image_path)[:2]:
image = load_image(os.path.join(image_path, file))
prompts.append(('请详细描述图片内容。', image))

pipe = pipeline(
model_path,
backend_config=PytorchEngineConfig(session_len=16384, tp=2),
chat_template_config=ChatTemplateConfig(model_name='internvl2_5')
)
response = pipe(prompts)`

ERROR:

`
2025-04-21 10:43:58,013 - lmdeploy - ERROR - engine.py:781 - Task failed
Traceback (most recent call last):
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 827, in async_loop
await self._async_loop_main(resp_que=resp_que, has_runable_event=has_runable_event)
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 762, in _async_loop_main
out = await self.executor.get_output_async()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 339, in get_output_async
return await self.remote_outs.get()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/asyncio/queues.py", line 159, in get
await getter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 776, in __task_callback
task.result()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 829, in async_loop
self._loop_finally()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 793, in _loop_finally
self.executor.release()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 308, in release
self.collective_rpc('exit')
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 243, in collective_rpc
return ray.get([getattr(worker, method).remote(*args, **kwargs) for worker in self.workers], timeout=timeout)
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 243, in
return ray.get([getattr(worker, method).remote(*args, **kwargs) for worker in self.workers], timeout=timeout)
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/ray/actor.py", line 1549, in __getattr__
raise AttributeError(
AttributeError: 'ActorHandle' object has no attribute 'exit'
unhandled exception during worker thread shutdown
task: exception=AttributeError("'ActorHandle' object has no attribute 'exit'")>
Traceback (most recent call last):
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 827, in async_loop
await self._async_loop_main(resp_que=resp_que, has_runable_event=has_runable_event)
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 762, in _async_loop_main
out = await self.executor.get_output_async()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 339, in get_output_async
return await self.remote_outs.get()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/asyncio/queues.py", line 159, in get
await getter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 776, in __task_callback
task.result()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 829, in async_loop
self._loop_finally()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 793, in _loop_finally
self.executor.release()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 308, in release
self.collective_rpc('exit')
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 243, in collective_rpc
return ray.get([getattr(worker, method).remote(*args, **kwargs) for worker in self.workers], timeout=timeout)
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 243, in
return ray.get([getattr(worker, method).remote(*args, **kwargs) for worker in self.workers], timeout=timeout)
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/ray/actor.py", line 1549, in __getattr__
raise AttributeError(
AttributeError: 'ActorHandle' object has no attribute 'exit'
(RayWorkerWrapper pid=451466) loc("/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":207:11): error: operation scheduled before its operands
Future exception was never retrieved
future:
Traceback (most recent call last):
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 827, in async_loop
await self._async_loop_main(resp_que=resp_que, has_runable_event=has_runable_event)
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 762, in _async_loop_main
out = await self.executor.get_output_async()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 339, in get_output_async
return await self.remote_outs.get()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/asyncio/queues.py", line 159, in get
await getter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 776, in __task_callback
task.result()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 829, in async_loop
self._loop_finally()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 793, in _loop_finally
self.executor.release()
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 308, in release
self.collective_rpc('exit')
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 243, in collective_rpc
return ray.get([getattr(worker, method).remote(*args, **kwargs) for worker in self.workers], timeout=timeout)
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 243, in
return ray.get([getattr(worker, method).remote(*args, **kwargs) for worker in self.workers], timeout=timeout)
File "/home/ai-admin/.conda/envs/lmdeploy/lib/python3.10/site-packages/ray/actor.py", line 1549, in __getattr__
raise AttributeError(
AttributeError: 'ActorHandle' object has no attribute 'exit'
`