Could not find any implementation for node failure of TensorRT 8.6.13 when running MHA(multi head attention) on GPU A10

Description

I tried to convert my model to the engine using tensorrt. My model has a lot of cross-attention structures, similar to Transformer. First of all, with precision fp32, it runs without any problems:

trtexec --onnx=A.onnx 
    --saveEngine=A.trt 
	--memPoolSize=workspace:32768 
	--useCudaGraph 
    --tacticSources=-CUBLAS,-CUBLAS_LT,-CUDNN,-JIT_CONVOLUTIONS,-EDGE_MASK_CONVOLUTIONS 
	--profilingVerbosity=detailed 
	--separateProfileRun 
	--dumpLayerInfo 
	--verbose 
	2>&1 | tee A.log

Then I tried to reduce it to fp16 to speed up. In this case, I need to add the --disableMHA command. Otherwise, an error will be reported:

[10/10/2024-14:25:30] [V] [TRT] --------------- Timing Runner:  {ForeignNode[/my/node]}.(Myelin[0x80000023])
nvrtc_compile.cpp:940: CHECK(false) failed. NVRTC Compilation failure
[10/10/2024-14:37:09] [V] [TRT] Skipping tactic 0x0000000000000000 due to exception No Myelin Error exists
[10/10/2024-14:37:09] [V] [TRT]  {ForeignNode[/my/node]}(Myelin[0x80000023]) profiling completed in 699.229 seconds. Fastest Tactic: 0xd15ea5edd15ea5ed Time: inf
[10/10/2024-14:37:09] [E] Error[10]: Could not find any implementation for node  {ForeignNode[/my/node]}
[10/10/2024-14:37:10] [E] Error[10]: [optimizer.cpp::computeCosts::3897] Error Code 10: Internal Error (Could not find any implementation for node  {ForeignNode[/my/node]})
[10/10/2024-14:37:10] [E] Engine could not be created from network
[10/10/2024-14:37:10] [E] Error[10]: [optimizer.cpp::computeCosts::3897] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[/my/node]}.)

I changed the name of the error node to my/node for privacy, which has nothing to do with my problem.

I think this is because there is no FP16-precision MHA implementation in tensorrt's base plugin library.I looked up tensorrt's github and found the MHA implementation in the plugin/bertQKVToContextPlugin/fused_multihead_attention_v2 folder. For example, fused_multihead_attention_v2_fp16_64_64_kernel.sm90.cpp. so I put the libnvinfer_plugin.so file from the lib folder in the tensorrt installation path into the trtexec directive as a plugin:

trtexec --onnx=A.onnx 
    --saveEngine=A.trt 
	--plugins=/cudalib/third-part/tensorrt/x64-linux/lib/libnvinfer_plugin.so 
    --fp16 
	--memPoolSize=workspace:32768 
	--useCudaGraph 
    --tacticSources=-CUBLAS,-CUBLAS_LT,-CUDNN,-JIT_CONVOLUTIONS,-EDGE_MASK_CONVOLUTIONS 
	--profilingVerbosity=detailed 
	--separateProfileRun 
	--dumpLayerInfo 
	--verbose 
	2>&1 | tee A.log

So the terminal log is going to look like this：

[10/10/2024-14:23:23] [I] Loading standard plugins
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::CoordConvAC version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::CropAndResizeDynamic version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::DecodeBbox3DPlugin version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::GenerateDetection_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 2
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::ModulatedDeformConv2d version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::NMSDynamic_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::PillarScatterPlugin version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::ProposalDynamic version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::Proposal version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::ROIAlign_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::ScatterND version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::Split version 1
[10/10/2024-14:23:23] [V] [TRT] Registered plugin creator - ::VoxelGeneratorPlugin version 1
[10/10/2024-14:23:23] [I] Loading supplied plugin library: /cudalib/third-part/tensorrt/x64-linux/lib/libnvinfer_plugin.so
[10/10/2024-14:23:23] [I] [TRT] [MemUsageChange] Init CUDA: CPU +272, GPU +0, now: CPU 280, GPU 329 (MiB)
[10/10/2024-14:23:24] [V] [TRT] Trying to load shared library libnvinfer_builder_resource.so.8.6.13
[10/10/2024-14:23:24] [V] [TRT] Loaded shared library libnvinfer_builder_resource.so.8.6.13
[10/10/2024-14:23:29] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +1112, GPU +268, now: CPU 1432, GPU 597 (MiB)
[10/10/2024-14:23:29] [I] Start parsing network model.

Unfortunately, the run gets the same error.

I want to know, why did this mistake happen? Is libnvinfer_plugin.so called correctly?
What should I do if I want to accelerate the MHA structure in my model?

Thanks a lot

Environment

TensorRT Version:8.6.13

NVIDIA GPU:A10

NVIDIA Driver Version:

CUDA Version:12.3

CUDNN Version:

Start with the TensorRT 8.6.13 reproduction using trtexec, the A10, CUDA 12.3, FP16, and the --disableMHA difference. Inspect plugin/bertQKVToContextPlugin/fused_multihead_attention_v2 and the supplied libnvinfer_plugin.so loading log, then compare the failing MHA path with the plugin behavior. Done means explaining the missing implementation or identifying the supported way to accelerate this model.