PaddlePaddle / PaddlePaddle/PaddleX

PaddleX模型剪裁报错:parallel_for failed: no kernel image is available for execution on the device

Open
#910 10 comments 0 reactions 1 assignee View on GitHub

@cuicheng01 is already working on this.

Since Feb 26, 2024.

Dominant language
Python
Stars
6.3k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

问题类型:模型训练
问题描述
PaddleX 1.3.11 版本,模型训练没有问题,但是在模型剪裁的时候报错了。报错内容如下:

creating index...
index created!
Paddle Version: 1.8.5
PaddleX Version: 1.3.11
num_classes = 1
W0702 08:14:17.499222 211 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 86, Driver API Version: 11.2, Runtime API Version: 10.0
W0702 08:14:17.501857 211 device_context.cc:260] device: 0, cuDNN Version: 8.1.
W0702 08:14:22.019361 211 operator.cc:187] gaussian_random raises an exception thrust::system::system_error, parallel_for failed: no kernel image is available for execution on the device
/usr/local/lib/python3.7/dist-packages/paddle/fluid/executor.py:1070: UserWarning: The following exception is not an EOF exception.
"The following exception is not an EOF exception.")
Traceback (most recent call last):
File "model_prune_paddlex1.3.11.py", line 59, in
use_vdl=True)
File "/usr/local/lib/python3.7/dist-packages/paddlex/cv/models/yolo_v3.py", line 174, in train
early_stop_patience, resume_checkpoint, False)
File "/usr/local/lib/python3.7/dist-packages/paddlex/cv/models/ppyolo.py", line 354, in train
resume_checkpoint=resume_checkpoint)
File "/usr/local/lib/python3.7/dist-packages/paddlex/cv/models/base.py", line 247, in net_initialize
self.exe.run(startup_prog)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/executor.py", line 1071, in run
six.reraise(*sys.exc_info())
File "/usr/local/lib/python3.7/dist-packages/six.py", line 719, in reraise
raise value
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/executor.py", line 1066, in run
return_merged=return_merged)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/executor.py", line 1154, in _run_impl
use_program_cache=use_program_cache)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/executor.py", line 1229, in _run_program
fetch_var_name)
RuntimeError: parallel_for failed: no kernel image is available for execution on the device

各个工具的版本号如下:
paddle2onnx 0.6
paddlehub 2.1.0
paddlenlp 2.0.5
paddlepaddle-gpu 1.8.5.post107
paddleslim 1.1.1
paddlex 1.3.11

电脑显卡:RTX3090,系统:Ubuntu18.04

====================
请在这里描述您在使用过程中的问题,如模型训练出错,建议贴上模型训练代码,以便开发人员分析,并快速响应

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.