Kaggle / Kaggle/docker-python

TF TensorRT misconfigured

Abierto
#1,323 0 comentarios 0 reacciones 0 asignados Ver en GitHub
bug help wanted
Lenguaje dominante
Python
Estrellas
2.7k
Forks
1k
Merge medio
7 d 14 h
PR fusionados (30 d)
2

Descripción

## 🐛 Bug

Tensorflow TensorRT seems to be wrongly linked.

### To Reproduce

On a few recent images (including `gcr.io/kaggle-gpu-images/python latest 311277776c9b 7 days ago 47.2GB`) I see very different linked and loaded TensorRT libs, namely **8.4 vs 8.6**.
```python
import tensorflow.compiler as tf_cc
linked_trt_ver=tf_cc.tf2tensorrt._pywrap_py_utils.get_linked_tensorrt_version()
print(f"Linked TRT ver: {linked_trt_ver}")
loaded_trt_ver=tf_cc.tf2tensorrt._pywrap_py_utils.get_loaded_tensorrt_version()
print(f"Loaded TRT ver: {loaded_trt_ver}")
# Linked TRT ver: (8, 4, 3)
# Loaded TRT ver: (8, 6, 1)
```
This has been Python. Now, the system inference libraries are indeed at 8.6:
```
dpkg -l | grep TensorRT
```
Now, [minimal compatibility rules](https://github.com/tensorflow/tensorflow/blob/0ac768e43c17837d6deb685d2a47baf6b4db5857/tensorflow/python/compiler/tensorrt/trt_convert.py#L257-L267) are fulfilled - the loaded version more recent.

However, the linking doesn't work properly. Under these recent containers, minimal TensorRT samples crash:
```python
import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt
from tensorflow.python.saved_model import signature_constants
from tensorflow.python.saved_model import tag_constants

from tensorflow.keras.applications.resnet50 import ResNet50
tf_model_dir = './models/tf_model'
model = ResNet50(include_top=50, weights='imagenet')
model.save(tf_model_dir)

converter = trt.TrtGraphConverterV2(
input_saved_model_dir=tf_model_dir,
)
converter.convert()

MAX_BATCH_SIZE=1
def input_fn():
img = tf.random.normal((MAX_BATCH_SIZE, 224,224,3),dtype=tf.float32)
return (img, )

import faulthandler
faulthandler.enable()
converter.build(input_fn=input_fn) #SEGMENTATION FAULT can happen under missconfigured software!
```

### Expected behavior

Align versions and make the sample code runnable.

### Additional context

See the [NVIDIA installation guidelines](https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-700/pdf/TensorRT-Installation-Guide.pdf)

Conditions from `tensorrt
```python
"Loaded TensorRT %s but linked TensorFlow against TensorRT %s. A few "
"requirements must be met:\n"
"\t-It is required to use the same major version of TensorRT during "
"compilation and runtime.\n"
"\t-TensorRT does not support forward compatibility. The loaded "
"version has to be equal or more recent than the linked version.",
```

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Comienza reproduciendo la discrepancia de versiones de TensorRT en la imagen de Kaggle referenciada, utilizando las comprobaciones de versiones vinculadas/cargadas en Python y el ejemplo mínimo de conversión de ResNet50. Inspecciona la configuración de TensorFlow TensorRT alrededor de trt_convert.py y los paquetes de TensorRT instalados en el contenedor. Se considera terminado cuando las versiones vinculadas y cargadas están alineadas y el ejemplo se ejecuta sin un segmentation fault.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
docker, python, tensorflow
Área
infrastructure
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.