TF TensorRT misconfigured
- Langage dominant
- Python
- Étoiles
- 2.7k
- Forks
- 1k
- Merge moyen
- 7 j 14 h
- PR mergées (30 j)
- 2
Description
## 🐛 Bug
Tensorflow TensorRT seems to be wrongly linked.
### To Reproduce
On a few recent images (including `gcr.io/kaggle-gpu-images/python latest 311277776c9b 7 days ago 47.2GB`) I see very different linked and loaded TensorRT libs, namely **8.4 vs 8.6**.
```python
import tensorflow.compiler as tf_cc
linked_trt_ver=tf_cc.tf2tensorrt._pywrap_py_utils.get_linked_tensorrt_version()
print(f"Linked TRT ver: {linked_trt_ver}")
loaded_trt_ver=tf_cc.tf2tensorrt._pywrap_py_utils.get_loaded_tensorrt_version()
print(f"Loaded TRT ver: {loaded_trt_ver}")
# Linked TRT ver: (8, 4, 3)
# Loaded TRT ver: (8, 6, 1)
```
This has been Python. Now, the system inference libraries are indeed at 8.6:
```
dpkg -l | grep TensorRT
```
Now, [minimal compatibility rules](https://github.com/tensorflow/tensorflow/blob/0ac768e43c17837d6deb685d2a47baf6b4db5857/tensorflow/python/compiler/tensorrt/trt_convert.py#L257-L267) are fulfilled - the loaded version more recent.
However, the linking doesn't work properly. Under these recent containers, minimal TensorRT samples crash:
```python
import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt
from tensorflow.python.saved_model import signature_constants
from tensorflow.python.saved_model import tag_constants
from tensorflow.keras.applications.resnet50 import ResNet50
tf_model_dir = './models/tf_model'
model = ResNet50(include_top=50, weights='imagenet')
model.save(tf_model_dir)
converter = trt.TrtGraphConverterV2(
input_saved_model_dir=tf_model_dir,
)
converter.convert()
MAX_BATCH_SIZE=1
def input_fn():
img = tf.random.normal((MAX_BATCH_SIZE, 224,224,3),dtype=tf.float32)
return (img, )
import faulthandler
faulthandler.enable()
converter.build(input_fn=input_fn) #SEGMENTATION FAULT can happen under missconfigured software!
```
### Expected behavior
Align versions and make the sample code runnable.
### Additional context
See the [NVIDIA installation guidelines](https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-700/pdf/TensorRT-Installation-Guide.pdf)
Conditions from `tensorrt
```python
"Loaded TensorRT %s but linked TensorFlow against TensorRT %s. A few "
"requirements must be met:\n"
"\t-It is required to use the same major version of TensorRT during "
"compilation and runtime.\n"
"\t-TensorRT does not support forward compatibility. The loaded "
"version has to be equal or more recent than the linked version.",
```
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Piste de recherche
Commencez par reproduire l'incompatibilité de versions de TensorRT dans l'image Kaggle référencée, en utilisant les vérifications des versions liées/chargées en Python et l'exemple minimal de conversion de ResNet50. Examinez la configuration de TensorFlow TensorRT autour de trt_convert.py ainsi que les paquets TensorRT installés dans le conteneur. C'est terminé lorsque les versions liées et chargées correspondent et que l'exemple s'exécute sans segmentation fault.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- docker, python, tensorflow
- Domaine
- infrastructure
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 35/100