How to remove signal and wait layer in the engine?

Description

Using trt llm to generate llama classification model. I have two similar script to generate engine, the first is raw scripts, the second is base on example/llama/build.sh script.
However, the second engine is slower than the first engine, so I dump engine layer, there are many signal and wait layer(as the images below show) in the second engine. It seems happen at type cast.
Any idea why generate signal and wait layer and how to work around this layer.

Environment

TensorRT Version:
9.3.0
NVIDIA GPU:
L20
NVIDIA Driver Version:
535.161.08
CUDA Version:
12.2
CUDNN Version:
8.9.6

Operating System:
Ubuntu 22.04.3 LTS
Python Version (if applicable):
3.10.12
Tensorflow Version (if applicable):
no
PyTorch Version (if applicable):
2.2.2
Baremetal or Container (if so, version):
container nvidia/cuda:12.1.0-devel-ubuntu22.04

Relevant Files

Model link:

Steps To Reproduce

Commands or scripts:

Have you tried the latest release?: no, there are high cost to upgrade trt version

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):

Compare the two engine-generation scripts with example/llama/build.sh, focusing on the dumped signal and wait layers and the reported type-cast area. Reproduce under TensorRT 9.3.0, CUDA 12.2, and the L20 environment; done means identifying why the layers differ and documenting a validated workaround.