Kaggle / Kaggle/docker-python

TF TensorRT misconfigured

未關閉
#1,323 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
bug help wanted
主要語言
Python
星號
2.7k
分支
1k
平均合併
7 天 14 小時
30 天內合併 PR
2

描述

## 🐛 Bug

Tensorflow TensorRT seems to be wrongly linked.

### To Reproduce

On a few recent images (including `gcr.io/kaggle-gpu-images/python latest 311277776c9b 7 days ago 47.2GB`) I see very different linked and loaded TensorRT libs, namely **8.4 vs 8.6**.
```python
import tensorflow.compiler as tf_cc
linked_trt_ver=tf_cc.tf2tensorrt._pywrap_py_utils.get_linked_tensorrt_version()
print(f"Linked TRT ver: {linked_trt_ver}")
loaded_trt_ver=tf_cc.tf2tensorrt._pywrap_py_utils.get_loaded_tensorrt_version()
print(f"Loaded TRT ver: {loaded_trt_ver}")
# Linked TRT ver: (8, 4, 3)
# Loaded TRT ver: (8, 6, 1)
```
This has been Python. Now, the system inference libraries are indeed at 8.6:
```
dpkg -l | grep TensorRT
```
Now, [minimal compatibility rules](https://github.com/tensorflow/tensorflow/blob/0ac768e43c17837d6deb685d2a47baf6b4db5857/tensorflow/python/compiler/tensorrt/trt_convert.py#L257-L267) are fulfilled - the loaded version more recent.

However, the linking doesn't work properly. Under these recent containers, minimal TensorRT samples crash:
```python
import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt
from tensorflow.python.saved_model import signature_constants
from tensorflow.python.saved_model import tag_constants

from tensorflow.keras.applications.resnet50 import ResNet50
tf_model_dir = './models/tf_model'
model = ResNet50(include_top=50, weights='imagenet')
model.save(tf_model_dir)

converter = trt.TrtGraphConverterV2(
input_saved_model_dir=tf_model_dir,
)
converter.convert()

MAX_BATCH_SIZE=1
def input_fn():
img = tf.random.normal((MAX_BATCH_SIZE, 224,224,3),dtype=tf.float32)
return (img, )

import faulthandler
faulthandler.enable()
converter.build(input_fn=input_fn) #SEGMENTATION FAULT can happen under missconfigured software!
```

### Expected behavior

Align versions and make the sample code runnable.

### Additional context

See the [NVIDIA installation guidelines](https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-700/pdf/TensorRT-Installation-Guide.pdf)

Conditions from `tensorrt
```python
"Loaded TensorRT %s but linked TensorFlow against TensorRT %s. A few "
"requirements must be met:\n"
"\t-It is required to use the same major version of TensorRT during "
"compilation and runtime.\n"
"\t-TensorRT does not support forward compatibility. The loaded "
"version has to be equal or more recent than the linked version.",
```

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

首先,使用 Python 中的連結/載入版本檢查以及最小的 ResNet50 轉換範例,重現所參照 Kaggle 映像中的 TensorRT 版本不相符問題。檢查 trt_convert.py 周圍的 TensorFlow TensorRT 設定,以及容器中安裝的 TensorRT 套件。當連結和載入的版本一致,且範例執行時沒有 segmentation fault,即表示完成。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
docker, python, tensorflow
領域
infrastructure
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。