Kaggle / Kaggle/docker-python

TF TensorRT misconfigured

Aperta
#1,323 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug help wanted
Lingua principale
Python
Stelle
2.7k
Fork
1k
Merge medio
7g 14h
PR unite (30g)
2

Descrizione

## 🐛 Bug

Tensorflow TensorRT seems to be wrongly linked.

### To Reproduce

On a few recent images (including `gcr.io/kaggle-gpu-images/python latest 311277776c9b 7 days ago 47.2GB`) I see very different linked and loaded TensorRT libs, namely **8.4 vs 8.6**.
```python
import tensorflow.compiler as tf_cc
linked_trt_ver=tf_cc.tf2tensorrt._pywrap_py_utils.get_linked_tensorrt_version()
print(f"Linked TRT ver: {linked_trt_ver}")
loaded_trt_ver=tf_cc.tf2tensorrt._pywrap_py_utils.get_loaded_tensorrt_version()
print(f"Loaded TRT ver: {loaded_trt_ver}")
# Linked TRT ver: (8, 4, 3)
# Loaded TRT ver: (8, 6, 1)
```
This has been Python. Now, the system inference libraries are indeed at 8.6:
```
dpkg -l | grep TensorRT
```
Now, [minimal compatibility rules](https://github.com/tensorflow/tensorflow/blob/0ac768e43c17837d6deb685d2a47baf6b4db5857/tensorflow/python/compiler/tensorrt/trt_convert.py#L257-L267) are fulfilled - the loaded version more recent.

However, the linking doesn't work properly. Under these recent containers, minimal TensorRT samples crash:
```python
import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt
from tensorflow.python.saved_model import signature_constants
from tensorflow.python.saved_model import tag_constants

from tensorflow.keras.applications.resnet50 import ResNet50
tf_model_dir = './models/tf_model'
model = ResNet50(include_top=50, weights='imagenet')
model.save(tf_model_dir)

converter = trt.TrtGraphConverterV2(
input_saved_model_dir=tf_model_dir,
)
converter.convert()

MAX_BATCH_SIZE=1
def input_fn():
img = tf.random.normal((MAX_BATCH_SIZE, 224,224,3),dtype=tf.float32)
return (img, )

import faulthandler
faulthandler.enable()
converter.build(input_fn=input_fn) #SEGMENTATION FAULT can happen under missconfigured software!
```

### Expected behavior

Align versions and make the sample code runnable.

### Additional context

See the [NVIDIA installation guidelines](https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-700/pdf/TensorRT-Installation-Guide.pdf)

Conditions from `tensorrt
```python
"Loaded TensorRT %s but linked TensorFlow against TensorRT %s. A few "
"requirements must be met:\n"
"\t-It is required to use the same major version of TensorRT during "
"compilation and runtime.\n"
"\t-TensorRT does not support forward compatibility. The loaded "
"version has to be equal or more recent than the linked version.",
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia riproducendo la discrepanza tra le versioni di TensorRT nell'immagine Kaggle indicata, utilizzando i controlli delle versioni collegate/caricate in Python e l'esempio minimo di conversione di ResNet50. Esamina la configurazione TensorFlow TensorRT relativa a trt_convert.py e i pacchetti TensorRT installati nel container. Il lavoro è completato quando le versioni collegate e caricate sono allineate e l'esempio viene eseguito senza un segmentation fault.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
docker, python, tensorflow
Ambito
infrastructure
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.