Different numbers of outputs result in inconsistent running results (original: 不同个数的输出导致运行结果不一致)
@yuanyao-nv is already working on this.
Since Feb 11, 2025.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Description
Auto-translated
(Part of the linear attention code)
hidden_states = query_KV / query_Z
return hidden_states, query_KV, query_Z
and
hidden_states = query_KV / query_Z
return hidden_states
When converting to TensorRT from ONNX, the results are different between the two above methods. The first is correct, while the second results in NaN; it seems that outputting the intermediate states affects the correctness of the results. How can this issue be resolved?
Further testing shows: This issue occurs with TensorRT 10.7, but TensorRT 10.6 works correctly.
Original
(部分linear attetion代码)
hidden_states = query_KV / query_Z
return hidden_states, query_KV, query_Z
和
hidden_states = query_KV / query_Z
return hidden_states
上面两者方式onnx转tensorrt时,两者结果不一样,前者是正确的,后者会出现nan;
这相当于输出了中间状态会导致结果的正确性,该怎么解决这种问题哇?
后面测试:tensorrt 10.7会出现这种问题,10.6是正确的
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.