[Bug]: Llama-3-70B FP16/BF16 inference fails while FP8 works on TensorRT-LLM 0.21.c-rc0 with 8x RTX 6000D

Description

I am testing Llama-3-70B inference performance with TensorRT-LLM 0.21.c-rc0 on an 8-GPU NVIDIA RTX 6000D system.

The target workload is:

• Input length: 128 tokens
• Output length: 2048 tokens
• Concurrency: 1 and 4
• Tensor Parallelism (TP): 8
• Pipeline Parallelism (PP): 1

During testing, I found that the behavior is different between FP8 quantization and non-quantized inference.

---

Environment

Hardware

• GPU:
  • 8 x NVIDIA RTX 6000D

Model

• Model:
  • Meta-Llama-3-70B

TensorRT-LLM Version

• TensorRT-LLM:
  • 0.21.c-rc0

Parallel Configuration

• Tensor Parallelism (TP): 8
• Pipeline Parallelism (PP): 1

Quantization

Two configurations were tested:

1. FP8 quantization
2. Without quantization (FP16/BF16)

Software Environment

• OS:debian 12
• CUDA version:13.0
• NVIDIA Driver version:580.82.07
• TensorRT version:0.21.0rc0

---

Issue Description

Case 1: FP8 quantization

The inference runs successfully.

Configuration:

• Model: Llama-3-70B
• GPU: 8 x NVIDIA RTX 6000D
• TP=8
• PP=1
• FP8 quantization

Test results:

Concurrency	Result
1	Successful
4	Successful

---

Case 2: Without quantization

Using the same hardware, model, and parallel configuration, but without quantization, the inference fails.

Configuration:

• Model: Llama-3-70B
• GPU: 8 x NVIDIA RTX 6000D
• TP=8
• PP=1
• FP16/BF16

Test results:

Concurrency	Result
1	Successful
4	Failed

The failure occurs during runtime execution.

Please see the attached logs:

• Attachment 1:
  • Non-quantized inference failure log

non-quantized.log

• Attachment 2:
  • FP8 quantization successful execution log

fp8-quantization.log

---

Benchmark Configuration

input_len=128
output_len=2048

concurrency=1   # or 4

TP=8
PP=1

No source file or test is named. Start by reproducing the reported Meta-Llama-3-70B configuration with TP=8, PP=1 and concurrency 4, then compare non-quantized.log with the successful FP8 log; done means identifying and fixing the runtime failure while preserving the reported successful cases.