NVIDIA / NVIDIA/TensorRT

The CustomQKVToContextPluginDynamic plugin uses unsatisfactory results

Open
#2,982 3 comments 0 reactions 1 assignee View on GitHub

@ttyio is already working on this.

Since May 18, 2023.

triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

I was trying to use the CustomQKVToContextPluginDynamic plugin plugin and found incorrect results. I don't know what caused it.

The input I gave is as follows:
q_tensor [4 4 4 4] [S B E]
k_tensor [6 6 6 6] [S B E]
v_tensor [4 4 4 4] [S B E]

The result of the plugin is not the same as Torch.Is there anything wrong with my operation?
The correct result returned should be [4 4 4 4]
But what the plugin returns is
trt: [[[[[6.]]
[[4.]]

[[6.]]

[[6.]]]]]

Environment

TensorRT Version 8.4.11:

NVIDIA GPU A40:

NVIDIA Driver Version:

CUDA Version:

CUDNN Version:

Operating System:

Python Version (if applicable) 3.9:

Tensorflow Version (if applicable):

PyTorch Version (if applicable) 2.0:

The code is as follows:
https://drive.google.com/file/d/1zVklW5pBDfgEVMsBzsxS5kBfTMm1oLJg/view?usp=sharing
https://drive.google.com/file/d/1c5vSCeho2GSnxVel-fNHzh7T_3jL4twF/view?usp=sharing

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.