I Got under 1.0 RPS when i run through TensorRT

Description

I got under 1.0 RPS when i run through TensorRT
Is it normal?
Please check below

Environment

TensorRT Version: tritonserver:25.08-trtllm-python-py3

NVIDIA GPU: H100 * 2

NVIDIA Driver Version: v12.2

CUDA Version: 535.86.10

CUDNN Version:

Relevant Files

Model link: private model

1. TensorRT-LLM Build
   trtllm-build --checkpoint_dir /mnt/models/models/trtllm/PrivateModel-37B-Instruct-250819/h100-2-gpu/quant/ \
   --output_dir /mnt/models/models/trtllm/PrivateModel-37B-Instruct-250819/h100-2-gpu/engine/ \
   --max_input_len 32768 \
   --max_batch_size 32
   --kv_cache_type paged
   --gemm_plugin auto
   --workers 2

2)modify_config.sh
ENGINE_DIR=/mnt/models/models/trtllm/PrivateModel-37B-Instruct-250819/h100-2-gpu/engine
TOKENIZER_DIR=/mnt/models/models/PrivateModel-37B-Instruct-250819
MODEL_FOLDER=/mnt/models/models/trtllm/PrivateModel-37B-Instruct-250819/model_repo
TRITON_MAX_BATCH_SIZE=4
INSTANCE_COUNT=1
MAX_QUEUE_DELAY_MS=0
MAX_QUEUE_SIZE=0
FILL_TEMPLATE_SCRIPT=/app/tools/fill_template.py
DECOUPLED_MODE=false
LOGITS_DATATYPE=TYPE_FP32

python3 ${FILL_TEMPLATE_SCRIPT} -i ${MODEL_FOLDER}/ensemble/config.pbtxt triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},logits_datatype:${LOGITS_DATATYPE}
python3 ${FILL_TEMPLATE_SCRIPT} -i ${MODEL_FOLDER}/preprocessing/config.pbtxt tokenizer_dir:${TOKENIZER_DIR},triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},preprocessing_instance_count:${INSTANCE_COUNT}
python3 ${FILL_TEMPLATE_SCRIPT} -i ${MODEL_FOLDER}/tensorrt_llm/config.pbtxt triton_backend:tensorrtllm,triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},decoupled_mode:${DECOUPLED_MODE},engine_dir:${ENGINE_DIR},max_queue_delay_microseconds:${MAX_QUEUE_DELAY_MS},batching_strategy:inflight_fused_batching,max_queue_size:${MAX_QUEUE_SIZE},encoder_input_features_data_type:TYPE_FP16,logits_datatype:${LOGITS_DATATYPE}
python3 ${FILL_TEMPLATE_SCRIPT} -i ${MODEL_FOLDER}/postprocessing/config.pbtxt tokenizer_dir:${TOKENIZER_DIR},triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},postprocessing_instance_count:${INSTANCE_COUNT}
python3 ${FILL_TEMPLATE_SCRIPT} -i ${MODEL_FOLDER}/tensorrt_llm_bls/config.pbtxt triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},decoupled_mode:${DECOUPLED_MODE},bls_instance_count:${INSTANCE_COUNT},logits_datatype:${LOGITS_DATATYPE}

3)serving (using kserve)
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: trtllm-privatemodel-fp8
namespace: russianblue
spec:
predictor:
containers:
- command: ["/bin/bash", "-c"]
args:
- |
python3 /app/scripts/launch_triton_server.py --world_size=2 --model_repo=/mnt/models/models/trtllm/PrivateModel-37B-Instruct-250819/model_repo --http_port=8080 & tail -f /dev/null &
while true; do
curl -X POST -H "Content-Type: text/plain" --data-binary @- http://prometheus-oss-pushgateway.oss.svc.cluster.local:9091/metrics/job/trt/instance/trtllm-privatemodel-fp8 <<< "$(curl -s localhost:8002/metrics)"
sleep 3;
done
image: aspcr01-queffmyz.scr.skr-west.scp-in.com/serving/tritonserver:25.08-trtllm-python-py3
name: kserve-container
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
cpu: "24"
memory: 128Gi
nvidia.com/gpu: "2"
requests:
cpu: "1"
memory: 2Gi
volumeMounts:
- mountPath: /dev/shm
name: shm
- mountPath: /mnt/models
name: model-volume
maxReplicas: 1
minReplicas: 1
volumes:
- emptyDir:
medium: Memory
sizeLimit: 50Gi
name: shm
- name: model-volume
persistentVolumeClaim:
claimName: llm-volume

Start by reviewing the supplied trtllm-build command, modify_config.sh settings, and KServe container command, then reproduce the serving setup if the private model and environment are available. Compare the observed RPS with an appropriate benchmark and document whether the result is expected; the private model and missing follow-up make completion difficult to verify.