deepjavalibrary / deepjavalibrary/djl

djl runs failure with dynamic input format

Open
#2,294 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4.9k
Forks
759
Avg merge
16h 15m
Merged PRs (30d)
15

Description

## Description

Use same onnx model as input to transform to tensorrt model with trtexec,if fix input format djl runs ok:

```
TensorRT-8.4.1.5/bin/trtexec --onnx=models/model.onnx --shapes=input_ids:1x4 --fp16 --saveEngine=model8415-1*4.trt
```

output:
```
DEBUG [main] 2023-01-05 00:47:05 Registering EngineProvider: TensorRT
DEBUG [main] 2023-01-05 00:47:05 Registering EngineProvider: TensorFlow
DEBUG [main] 2023-01-05 00:47:05 Registering EngineProvider: MXNet
DEBUG [main] 2023-01-05 00:47:05 Found default engine: MXNet
DEBUG [main] 2023-01-05 00:47:05 Loading TensorRT JNI library from: /root/.djl.ai/tensorrt/8.4.1-0.19.0-linux-x86_64/libdjl_trt.so
DEBUG [main] 2023-01-05 00:47:05 Scanning models in repo: class ai.djl.repository.SimpleRepository, file:/tensorrt/model8415-1*4.trt
DEBUG [main] 2023-01-05 00:47:05 Loading model with Criteria:
Application: UNDEFINED
Input: class djl.input.TensorRTInput
Output: class djl.output.TensorRTOutput
Engine: TensorRT
ModelZoo: ai.djl.localmodelzoo

DEBUG [main] 2023-01-05 00:47:05 Searching model in specified model zoo: ai.djl.localmodelzoo
WARN [main] 2023-01-05 00:47:05 Simple repository pointing to a non-archive file.
DEBUG [main] 2023-01-05 00:47:05 Checking ModelLoader: ai.djl.localmodelzoo:model8415-1*4.trt UNDEFINED [
ai.djl.localmodelzoo/model8415-1*4.trt/model8415-1*4.trt {}
]
DEBUG [main] 2023-01-05 00:47:05 Preparing artifact: file:/tensorrt/model8415-1*4.trt, ai.djl.localmodelzoo/model8415-1*4.trt/model8415-1*4.trt {}
DEBUG [main] 2023-01-05 00:47:05 Skip prepare for local repository.
Loading: 100% |████████████████████████████████████████|
DEBUG [main] 2023-01-05 00:47:05 Using cache dir: /root/.djl.ai/mxnet/1.9.1-cu114mkl-linux-x86_64
DEBUG [main] 2023-01-05 00:47:05 Loading mxnet library from: /root/.djl.ai/mxnet/1.9.1-cu114mkl-linux-x86_64/libmxnet.so
DEBUG [main] 2023-01-05 00:47:07 Using cache dir: /root/.djl.ai/tensorflow
DEBUG [main] 2023-01-05 00:47:07 Loading TensorFlow library from: /root/.djl.ai/tensorflow/2.7.4-cu114-linux-x86_64/libjnitensorflow.so
DEBUG [main] 2023-01-05 00:47:07 Loading TensorRT UFF model /tensorrt/model8415-1*4.trt with options:
[TRT] INFO: [MemUsageChange] Init CUDA: CPU +273, GPU +0, now: CPU 1200, GPU 491 (MiB)
[TRT] INFO: Loaded engine size: 681 MiB
[TRT] INFO: [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +680, now: CPU 0, GPU 680 (MiB)
[TRT] INFO: [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +2, now: CPU 0, GPU 682 (MiB)
DEBUG [main] 2023-01-05 00:47:11 Model information:
DEBUG [main] 2023-01-05 00:47:11 input_0[input_ids]: int32, (1, 4)
DEBUG [main] 2023-01-05 00:47:11 output_0[output]: float32, (1, 4, 51200)
load model success
result size: 1
result: output: (4, 51200) cpu() float32
[ Exceed max print size ]
[[F@f237ae7, [F@42edde25, [F@6fe5da76, [F@77d95e5a]

```

dynamic input format runs failure core dump with message "Cuda failure: 2", but without core dump file:

```
TensorRT-8.4.1.5/bin/trtexec --onnx=models/model.onnx --minShapes=input_ids:1x1 --maxShapes=input_ids:1x16 --optShapes=input_ids:1x4 --workspace=3072 --fp16 --saveEngine=model8415-dynamic.trt
```

output:
```
DEBUG [main] 2023-01-05 02:58:57 Scanning models in repo: class ai.djl.repository.SimpleRepository, file:/tensorrt/model8415-dynamic.trt
DEBUG [main] 2023-01-05 02:58:57 Loading model with Criteria:
Application: UNDEFINED
Input: class djl.input.TensorRTInput
Output: class djl.output.TensorRTOutput
Engine: TensorRT
ModelZoo: ai.djl.localmodelzoo

DEBUG [main] 2023-01-05 02:58:57 Searching model in specified model zoo: ai.djl.localmodelzoo
DEBUG [main] 2023-01-05 02:58:57 Registering EngineProvider: TensorRT
DEBUG [main] 2023-01-05 02:58:57 Registering EngineProvider: TensorFlow
DEBUG [main] 2023-01-05 02:58:57 Registering EngineProvider: MXNet
DEBUG [main] 2023-01-05 02:58:57 Found default engine: MXNet
WARN [main] 2023-01-05 02:58:57 Simple repository pointing to a non-archive file.
DEBUG [main] 2023-01-05 02:58:57 Checking ModelLoader: ai.djl.localmodelzoo:model8415-dynamic.trt UNDEFINED [
ai.djl.localmodelzoo/model8415-dynamic.trt/model8415-dynamic.trt {}
]
DEBUG [main] 2023-01-05 02:58:57 Preparing artifact: file:/tensorrt/model8415-dynamic.trt, ai.djl.localmodelzoo/model8415-dynamic.trt/model8415-dynamic.trt {}
DEBUG [main] 2023-01-05 02:58:57 Skip prepare for local repository.
Loading: 100% |████████████████████████████████████████|
DEBUG [main] 2023-01-05 02:58:58 Loading TensorRT JNI library from: /root/.djl.ai/tensorrt/8.4.1-0.19.0-linux-x86_64/libdjl_trt.so
DEBUG [main] 2023-01-05 02:58:58 Using cache dir: /root/.djl.ai/mxnet/1.9.1-cu114mkl-linux-x86_64
DEBUG [main] 2023-01-05 02:58:58 Loading mxnet library from: /root/.djl.ai/mxnet/1.9.1-cu114mkl-linux-x86_64/libmxnet.so
DEBUG [main] 2023-01-05 02:58:59 Using cache dir: /root/.djl.ai/tensorflow
DEBUG [main] 2023-01-05 02:58:59 Loading TensorFlow library from: /root/.djl.ai/tensorflow/2.7.4-cu114-linux-x86_64/libjnitensorflow.so
DEBUG [main] 2023-01-05 02:58:59 Loading TensorRT UFF model /tensorrt/model8415-dynamic.trt with options:
[TRT] INFO: [MemUsageChange] Init CUDA: CPU +273, GPU +0, now: CPU 1538, GPU 491 (MiB)
[TRT] INFO: Loaded engine size: 1006 MiB
[TRT] INFO: [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +1001, now: CPU 0, GPU 1001 (MiB)
[TRT] INFO: [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +8, now: CPU 0, GPU 1009 (MiB)
Cuda failure: 2
Aborted (core dumped)

```
It is worth noting that model8415-dynamic.trt works fine use python tensorrt.

## Environment Info

cuda: 11.4
tensorrt: 8.4.1.5
os: ubuntu18.04

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.