Internal Error (Could not find any implementation for node {...Tensordot/Reshape}) failure of TensorRT 8.6 when running generate INT8 engine on GPU RTX3080
@yuanyao-nv is already working on this.
Since Dec 19, 2024.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Hi,
I'm using TensorRT8.6 to conduct int8 calibration and genereate tensorrt engine with polygraphy tools like this
polygraphy convert onnx_model/model.onnx --trt-min-shapes xs:[1,1120] xlen:[1] --trt-opt-shapes xs:[1,160000] xlen:[1] --trt-max-shapes xs:[1,480000] xlen:[1] --int8 --data-loader-script data_loader.py --calibration-cache trt86_minmax_calib.cache --calib-base-cls IInt8MinMaxCalibrator --output trt_model/trt86_minmax_int8.plan
, but it always give the following error:
[W] 'colored' module is not installed, will not use colors when logging. To enable colors, please install the 'colored' module: python3 -m pip install colored
[W] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[I] Configuring with profiles: [Profile().add('xs', min=[1, 1120], opt=[1, 160000], max=[1, 480000]).add('xlen', min=[1], opt=[1], max=[1])]
[W] TensorRT does not currently support using dynamic shapes during calibration. The `OPT` shapes from the calibration profile will be used for tensors with dynamic shapes. Calibration data is expected to conform to those shapes.
[I] Building engine with configuration:
Flags | [INT8]
Engine Capability | EngineCapability.DEFAULT
Memory Pools | [WORKSPACE: 10002.44 MiB, TACTIC_DRAM: 10002.44 MiB]
Tactic Sources | [CUBLAS, CUBLAS_LT, CUDNN, EDGE_MASK_CONVOLUTIONS, JIT_CONVOLUTIONS]
Profiling Verbosity | ProfilingVerbosity.DETAILED
Preview Features | [FASTER_DYNAMIC_SHAPES_0805, DISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805]
Calibrator | Calibrator(<generator object load_data at 0x7fd95cd5c430>, cache='trt86_minmax_calib.cache', BaseClass=<class 'tensorrt.tensorrt.IInt8MinMaxCalibrator'>)
[I] Saving calibration cache to trt86_minmax_calib.cache
[W] Missing scale and zero-point for tensor xlen, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor (Unnamed Layer* 1) [Shuffle]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor sub:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor (Unnamed Layer* 9) [Shuffle]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor floordiv:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor stft/frame/range_1:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor (Unnamed Layer* 59) [Shuffle]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor stft/frame/mul_1:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor stft/frame/Reshape_2:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor (Unnamed Layer* 70) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor stft/frame/add_2:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor (Unnamed Layer* 82) [Shuffle]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor add:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor SequenceMask/ExpandDims:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor SequenceMask/Cast:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor (Unnamed Layer* 107) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor (Unnamed Layer* 112) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor SequenceMask/Range:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[W] Missing scale and zero-point for tensor SequenceMask/Less:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
formats.cpp:2379: DCHECK(desired_so.size() == t->dim_count()) failed.
[E] 10: Could not find any implementation for node {ForeignNode[(Unnamed Layer* 82) [Shuffle]_output[Constant]...Tensordot/Reshape]}.
[E] 10: [optimizer.cpp::computeCosts::3869] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[(Unnamed Layer* 82) [Shuffle]_output[Constant]...Tensordot/Reshape]}.)
[!] Invalid Engine. Please ensure the engine was built correctly
The convert onnx model is generated from the saved model, which produced with codes as below:
import librosa
import tensorflow as tf
xs = tf.placeholder(tf.float32, shape=[None, None], name='xs')
xlen = tf.placeholder(tf.int32, shape=[None], name='xlen')
spectrogram = tf.square(tf.abs(tf.signal.stft(xs, frame_length=400, frame_step=160, fft_length=512)))
weight = librosa.filters.mel(sr=16000, n_fft=512, n_mels=80).T # (257,80)
weight = tf.convert_to_tensor(weight, tf.float32)
mel = tf.tensordot(spectrogram, weight, axes=1)
mel_length = (xlen - 400) // 160 + 1
mask = tf.sequence_mask(mel_length, maxlen=tf.shape(mel)[1], dtype=tf.float32)
mel = mel * tf.expand_dims(mask, axis=-1)
ys = tf.identity(mel, name='ys')
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
tf.saved_model.simple_save(sess,
'./saved_model',
inputs={"xs": xs, "xlen": xlen},
outputs={"ys": ys})
Then, convert saved model to onnx model with python3 -m tf2onnx.convert --opset 13 --saved-model ./saved_model/ --output ./onnx_model/model.onnx , what's wrong with it ?
By the way, the file data_loader.py is used for int8 calibration, you can reproduce it like this:
import numpy as np
def load_data(calib_num=100):
for _ in range(calib_num):
n = np.random.randint(1000, 5000)
x = np.random.rand(1, n).astype(np.float32)
x_len = np.array([n], dtype=np.int32)
yield {"xs": x, "xlen": x_len}
Anyone can give some helps? Thanks a lot for the help!!!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.