Could not find TensorRT, resulting inference without GPU

After installation, I was able to run inference in CPU mode, but it fails to run with GPU,

- I have tried to install `tensorrt` with pip,
- I have added its path to `LD_LIBARARY_PATH`,
- I have try to link the library with other name,

but I still fails to run tensorflow with GPU, not sure what am I missing here.

Here are the settings for my setup, I am using A100 with CUDA 11.8
```bash
(gpugeo) $ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2024-04-04 20:13:57.014520: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-04 20:13:58.875746: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-04-04 20:14:03.631155: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1960] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]
$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.6 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.6 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8"
(gpugeo) [APAC-TY@b05u17g geo]$ nvidia-smi
Thu Apr 4 20:14:14 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05 Driver Version: 520.61.05 CUDA Version: 11.8 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-SXM... On | 00000000:26:00.0 Off | 0 |
| N/A 24C P0 60W / 400W | 31MiB / 81920MiB | 0% E. Process |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA A100-SXM... On | 00000000:2C:00.0 Off | 0 |
| N/A 31C P0 206W / 400W | 3041MiB / 81920MiB | 46% E. Process |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 2 NVIDIA A100-SXM... On | 00000000:66:00.0 Off | 0 |
| N/A 37C P0 353W / 400W | 54879MiB / 81920MiB | 89% E. Process |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 3 NVIDIA A100-SXM... On | 00000000:6B:00.0 Off | 0 |
| N/A 24C P0 61W / 400W | 0MiB / 81920MiB | 0% E. Process |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 4 NVIDIA A100-SXM... On | 00000000:A4:00.0 Off | 0 |
| N/A 23C P0 60W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 5 NVIDIA A100-SXM... On | 00000000:A9:00.0 Off | 0 |
| N/A 24C P0 59W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 6 NVIDIA A100-SXM... On | 00000000:E1:00.0 Off | 0 |
| N/A 23C P0 61W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 7 NVIDIA A100-SXM... On | 00000000:E7:00.0 Off | 0 |
| N/A 23C P0 61W / 400W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 3812720 C nvidia-cuda-mps-server 28MiB |
| 1 N/A N/A 3744900 M+C lmp_mpi 3010MiB |
| 1 N/A N/A 3744976 C nvidia-cuda-mps-server 28MiB |
| 2 N/A N/A 3772436 M+C python 54848MiB |
| 2 N/A N/A 3772443 C nvidia-cuda-mps-server 28MiB |
+-----------------------------------------------------------------------------+
(gpugeo) $ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2024-04-04 20:14:20.371825: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-04 20:14:22.177861: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-04-04 20:14:26.929802: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1960] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
(gpugeo) $ conda list
# packages in environment at /work/APAC-TY/miniconda3/envs/gpugeo:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main defaults
_openmp_mutex 5.1 1_gnu defaults
absl-py 1.4.0 pypi_0 pypi
array-record 0.4.1 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
babel 2.13.0 pypi_0 pypi
beautifulsoup4 4.12.2 pypi_0 pypi
bzip2 1.0.8 h5eee18b_5 defaults
ca-certificates 2024.3.11 h06a4308_0 defaults
cached-property 1.5.2 pypi_0 pypi
cachetools 5.3.1 pypi_0 pypi
certifi 2023.7.22 pypi_0 pypi
charset-normalizer 3.3.0 pypi_0 pypi
chex 0.1.7 pypi_0 pypi
click 8.1.7 pypi_0 pypi
clu 0.0.7 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
commonmark 0.9.1 pypi_0 pypi
contextlib2 21.6.0 pypi_0 pypi
contourpy 1.1.1 pypi_0 pypi
cuda 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-cccl 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-command-line-tools 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-compiler 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-cudart 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-cudart-dev 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-cuobjdump 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-cupti 11.8.87 0 nvidia/label/cuda-11.8.0
cuda-cuxxfilt 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-demo-suite 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-documentation 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-driver-dev 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-gdb 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-libraries 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-libraries-dev 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-memcheck 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-nsight 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-nsight-compute 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-nvcc 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-nvdisasm 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-nvml-dev 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-nvprof 11.8.87 0 nvidia/label/cuda-11.8.0
cuda-nvprune 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-nvrtc 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-nvrtc-dev 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-nvtx 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-nvvp 11.8.87 0 nvidia/label/cuda-11.8.0
cuda-profiler-api 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-runtime 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-sanitizer-api 11.8.86 0 nvidia/label/cuda-11.8.0
cuda-toolkit 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-tools 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-visual-tools 11.8.0 0 nvidia/label/cuda-11.8.0
cycler 0.12.1 pypi_0 pypi
dm-tree 0.1.8 pypi_0 pypi
docstring-parser 0.15 pypi_0 pypi
editdistance 0.6.2 pypi_0 pypi
etils 1.5.1 pypi_0 pypi
filelock 3.12.4 pypi_0 pypi
flatbuffers 23.5.26 pypi_0 pypi
flax 0.5.3 pypi_0 pypi
fonttools 4.43.1 pypi_0 pypi
fsspec 2023.9.2 pypi_0 pypi
future 0.18.3 pypi_0 pypi
gast 0.4.0 pypi_0 pypi
gdown 4.7.1 pypi_0 pypi
gds-tools 1.4.0.31 0 nvidia/label/cuda-11.8.0
gin 0.1.6 pypi_0 pypi
gin-config 0.5.0 pypi_0 pypi
google-auth 2.23.3 pypi_0 pypi
google-auth-oauthlib 1.0.0 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
googleapis-common-protos 1.61.0 pypi_0 pypi
grpcio 1.59.0 pypi_0 pypi
h5py 3.10.0 pypi_0 pypi
huggingface-hub 0.17.3 pypi_0 pypi
idna 3.4 pypi_0 pypi
immutabledict 3.0.0 pypi_0 pypi
importlib-resources 6.1.0 pypi_0 pypi
jax 0.4.6 pypi_0 pypi
jaxlib 0.4.6 pypi_0 pypi
joblib 1.3.2 pypi_0 pypi
keras 2.13.1 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1 defaults
libclang 16.0.6 pypi_0 pypi
libcublas 11.11.3.6 0 nvidia/label/cuda-11.8.0
libcublas-dev 11.11.3.6 0 nvidia/label/cuda-11.8.0
libcufft 10.9.0.58 0 nvidia/label/cuda-11.8.0
libcufft-dev 10.9.0.58 0 nvidia/label/cuda-11.8.0
libcufile 1.4.0.31 0 nvidia/label/cuda-11.8.0
libcufile-dev 1.4.0.31 0 nvidia/label/cuda-11.8.0
libcurand 10.3.0.86 0 nvidia/label/cuda-11.8.0
libcurand-dev 10.3.0.86 0 nvidia/label/cuda-11.8.0
libcusolver 11.4.1.48 0 nvidia/label/cuda-11.8.0
libcusolver-dev 11.4.1.48 0 nvidia/label/cuda-11.8.0
libcusparse 11.7.5.86 0 nvidia/label/cuda-11.8.0
libcusparse-dev 11.7.5.86 0 nvidia/label/cuda-11.8.0
libffi 3.3 he6710b0_2 defaults
libgcc-ng 11.2.0 h1234567_1 defaults
libgomp 11.2.0 h1234567_1 defaults
libnpp 11.8.0.86 0 nvidia/label/cuda-11.8.0
libnpp-dev 11.8.0.86 0 nvidia/label/cuda-11.8.0
libnvjpeg 11.9.0.86 0 nvidia/label/cuda-11.8.0
libnvjpeg-dev 11.9.0.86 0 nvidia/label/cuda-11.8.0
libstdcxx-ng 11.2.0 h1234567_1 defaults
libuuid 1.41.5 h5eee18b_0 defaults
lxml 4.9.3 pypi_0 pypi
markdown 3.5 pypi_0 pypi
markupsafe 2.1.3 pypi_0 pypi
matplotlib 3.7.0 pypi_0 pypi
mesh-tensorflow 0.1.21 pypi_0 pypi
ml-collections 0.1.1 pypi_0 pypi
msgpack 1.0.7 pypi_0 pypi
ncurses 6.4 h6a678d5_0 defaults
nltk 3.8.1 pypi_0 pypi
nsight-compute 2022.3.0.22 0 nvidia/label/cuda-11.8.0
numpy 1.23.5 pypi_0 pypi
nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
nvidia-cudnn-cu12 9.0.0.312 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
openssl 1.1.1w h7f8727e_0 defaults
opt-einsum 3.3.0 pypi_0 pypi
optax 0.1.7 pypi_0 pypi
packaging 23.2 pypi_0 pypi
pandas 2.1.1 pypi_0 pypi
pillow 10.0.1 pypi_0 pypi
pip 24.0 pypi_0 pypi
portalocker 2.8.2 pypi_0 pypi
promise 2.3 pypi_0 pypi
protobuf 3.20.3 pypi_0 pypi
psutil 5.9.5 pypi_0 pypi
pyasn1 0.5.0 pypi_0 pypi
pyasn1-modules 0.3.0 pypi_0 pypi
pyglove 0.4.3 pypi_0 pypi
pygments 2.16.1 pypi_0 pypi
pyparsing 3.1.1 pypi_0 pypi
pysocks 1.7.1 pypi_0 pypi
python 3.10.0 h12debd9_5 defaults
python-dateutil 2.8.2 pypi_0 pypi
pytz 2023.3.post1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
readline 8.2 h5eee18b_0 defaults
regex 2023.10.3 pypi_0 pypi
requests 2.31.0 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
rich 11.2.0 pypi_0 pypi
rouge-score 0.1.2 pypi_0 pypi
rsa 4.9 pypi_0 pypi
sacrebleu 2.3.1 pypi_0 pypi
safetensors 0.4.0 pypi_0 pypi
scikit-learn 1.3.1 pypi_0 pypi
scipy 1.10.0 pypi_0 pypi
sentencepiece 0.1.99 pypi_0 pypi
seqio 0.0.18 pypi_0 pypi
seqio-nightly 0.0.17.dev20231013 pypi_0 pypi
setuptools 68.2.2 py310h06a4308_0 defaults
six 1.16.0 pypi_0 pypi
soupsieve 2.5 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0 defaults
t5 0.9.4 pypi_0 pypi
tabulate 0.9.0 pypi_0 pypi
tensorboard 2.13.0 pypi_0 pypi
tensorboard-data-server 0.7.1 pypi_0 pypi
tensorflow 2.13.0 pypi_0 pypi
tensorflow-datasets 4.9.3 pypi_0 pypi
tensorflow-estimator 2.13.0 pypi_0 pypi
tensorflow-hub 0.15.0 pypi_0 pypi
tensorflow-io-gcs-filesystem 0.34.0 pypi_0 pypi
tensorflow-metadata 1.14.0 pypi_0 pypi
tensorflow-text 2.13.0 pypi_0 pypi
tensorrt-libs 8.6.1 pypi_0 pypi
tensorstore 0.1.45 pypi_0 pypi
termcolor 2.3.0 pypi_0 pypi
tfds-nightly 4.9.2.dev202308090034 pypi_0 pypi
threadpoolctl 3.2.0 pypi_0 pypi
tk 8.6.12 h1ccaba5_0 defaults
tokenizers 0.14.1 pypi_0 pypi
toml 0.10.2 pypi_0 pypi
toolz 0.12.0 pypi_0 pypi
tqdm 4.66.1 pypi_0 pypi
transformers 4.34.0 pypi_0 pypi
typing-extensions 4.5.0 pypi_0 pypi
tzdata 2023.3 pypi_0 pypi
urllib3 2.0.6 pypi_0 pypi
werkzeug 3.0.0 pypi_0 pypi
wheel 0.41.2 py310h06a4308_0 defaults
wrapt 1.15.0 pypi_0 pypi
xz 5.4.6 h5eee18b_0 defaults
zipp 3.17.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_0 defaults
[]
```