When using the tensorrt api to build the bert base model, the inference result is Nan.
Open
@ttyio is already working on this.
Since Mar 5, 2023.
triaged
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Description
Environment
TensorRT Version: 8.5.1.7
NVIDIA GPU: V100
NVIDIA Driver Version: 470.103.01
CUDA Version: 11.8
I used bert_tf_ckpt_base_qa_squad2_amp_384_19.03.0 to build trt engine. If I set the model input sequence length to 384, the model inference result is normal, but if I change the sequence length to 128, the inference result is nan.
I found that after the attention layer called the CustomQKVToContextPluginDynamic plugin, the output result of attention layer is nan. I used bert-base-chinese to build trt engine, too. But the situation is same with bert_tf_ckpt_base_qa_squad2_amp_384_19.03.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.