Plenty of regressions in trt-llm v0.20.0

System Info

We have a range of configurations that used to build in trt 0.18.1 and does no longer build or run with the C++ Executor on trt 0.20.0rc0.

The only thing that has changed here is a dependency upgrade from trt 0.18.1 vs 0.20.0rc0

Example:
LLama-3B goes out of memory on a 16GB ram machine

s to 466 nodes requiring 905976832 bytes.
[04/29/2025-16:34:56] [TRT] [I] Total Activation Memory: 905976832 bytes
[04/29/2025-16:34:56] [TRT] [I] Total Weights Memory: 7280613888 bytes
[04/29/2025-16:34:56] [TRT] [I] Compiler backend is used during engine execution.
[04/29/2025-16:34:56] [TRT] [I] Engine generation completed in 62.5946 seconds.
[04/29/2025-16:34:56] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 1503 MiB, GPU 9949 MiB
/bin/bash: line 47:    29 Killed                  python3 -m builder.main --config /app/config.yaml
User GPU build job failed with status: FAILED

Llama-3-1B gets build on a L4 (16GB RAM, 24GB VRam)

 only tokenizers, configuration and file/data utilities can be used.

[Coldboost] model_cache: Fetch took 1.09 seconds, of which 0.00 seconds were spent blocking.

[Coldboost] Using hot_reload: /usr/local/briton/bin/libhot_reload.so

[Coldboost] Waiting for Briton server for briton to start

[Coldboost] Waiting for Briton server for briton to start

Model terminated unexpectedly. This model does not have enough resources to run. Try upgrading to a larger instance type. Exit code: 137, reason: OOMKilled, restart count: 4

Model terminated unexpectedly. This model does not have enough resources to run. Try upgrading to a larger instance type. Exit code: 137, reason: OOMKilled, restart count: 5

Model terminated unexpectedly. This model does not have enough resources to run. Try upgrading to a larger instance type. Exit code: 137, reason: OOMKilled, restart count: 5

Llama-70B-H100:

Model terminated unexpectedly. This model does not have enough resources to run. Try upgrading to a larger instance type. Exit code: 137, reason: OOMKilled, restart count: 1

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Building (pretty similar to the above) and running with C++ Executor API.
Important to not that only dependencies have changed. I am sure this could be reporduced with the nvidia internal / trtion-like runtime. This is running on Baseten.co

Expected behavior

no regressions over trt 18.1

actual behavior

additional notes

Start by reproducing the reported builds and runs with TensorRT-LLM 0.18.1 and 0.20.0rc0 through the C++ Executor API, using the listed Llama configurations and memory logs. Compare dependency and resource behavior across versions; done means the regressions are isolated and corrected or clearly documented with a reproducible failure.