load unet.onnx failure of TensorRT 8.6 when running engine_from_network on GPU rtx4090
Open
@zerollzeng is already working on this.
Since Aug 24, 2023.
triaged
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Description
i use the demo_txt2img.py in demo/Diffusion,it works when using default arguments. but when i use the dynamic shape ,it fails with the error below:
[I] Configuring with profiles: [Profile().add('sample', min=(2, 4, 32, 32), opt=(2, 4, 64, 64), max=(8, 4, 128, 128)).add('encoder_hidden_states', min=(2, 77, 768), opt=(2, 77, 768), max=(8, 77, 768)).add('timestep', min=[1], opt=[1], max=[1])]
[I] Building engine with configuration:
Flags | [FP16]
Engine Capability | EngineCapability.DEFAULT
Memory Pools | [WORKSPACE: 16692.81 MiB, TACTIC_DRAM: 24217.31 MiB]
Tactic Sources | []
Profiling Verbosity | ProfilingVerbosity.DETAILED
Preview Features | [DISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805]
[W] Myelin graph with multiple dynamic values may have poor performance if they differ.
[E] 10: Could not find any implementation for node {ForeignNode[onnx::LayerNormalization_9097 + (Unnamed Layer* 1230) [Shuffle].../down_blocks.0/attentions.0/Reshape_1 + /down_blocks.0/attentions.0/Transpose_1]}.
[E] 10: [optimizer.cpp::computeCosts::3869] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[onnx::LayerNormalization_9097 + (Unnamed Layer* 1230) [Shuffle].../down_blocks.0/attentions.0/Reshape_1 + /down_blocks.0/attentions.0/Transpose_1]}.)
[!] Invalid Engine. Please ensure the engine was built correctly
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /media/74nvme/research/diffusers_tensorrt/convert/convert_diffusers_to_tensorrt.py:223 in │
│ <module> │
│ │
│ 220 │
│ 221 if __name__=="__main__": │
│ 222 │ args = parse() │
│ ❱ 223 │ convert2Engines(args) │
│ 224 │
│ 225 │
│ 226 │
│ │
│ /media/74nvme/research/diffusers_tensorrt/convert/convert_diffusers_to_tensorrt.py:205 in │
│ convert2Engines │
│ │
│ 202 │ │ │ except Exception as e: │
│ 203 │ │ │ │ print(e) │
│ 204 │ │ │ if args.force_engine_build or not os.path.exists(engine.engine_path): │
│ ❱ 205 │ │ │ │ engine.build(onnx_opt_path, │
│ 206 │ │ │ │ │ fp16=True, │
│ 207 │ │ │ │ │ input_profile=obj.get_input_profile( │
│ 208 │ │ │ │ │ │ args.opt_batch_size, args.opt_image_height, args.opt_image_width │
│ │
│ /media/74nvme/research/diffusers_tensorrt/convert/convert_diffusers_to_tensorrt.py:90 in build │
│ │
│ 87 │ │ if not enable_all_tactics: │
│ 88 │ │ │ config_kwargs['tactic_sources'] = [] │
│ 89 │ │ │
│ ❱ 90 │ │ engine = engine_from_network( │
│ 91 │ │ │ network_from_onnx_path(onnx_path), │
│ 92 │ │ │ config=CreateConfig(fp16=fp16, │
│ 93 │ │ │ │ refittable=enable_refit, │
│ in engine_from_network │
│ │
│ /home/arc-zjy3501/anaconda3/envs/sd-webui/lib/python3.10/site-packages/polygraphy/backend/base/l │
│ oader.py:40 in __call__ │
│ │
│ 37 │ │ Note: ``call_impl`` should *not* be called directly - use this function instead. │
│ 38 │ │ """ │
│ 39 │ │ __doc__ = self.call_impl.__doc__ │
│ ❱ 40 │ │ return self.call_impl(*args, **kwargs) │
│ 41 │
│ │
│ /home/arc-zjy3501/anaconda3/envs/sd-webui/lib/python3.10/site-packages/polygraphy/util/util.py:6 │
│ 94 in wrapped │
│ │
│ 691 │ │ │ │ │ │ f"Calling '{func.__qualname__}()' directly is not recommended. P │
│ 692 │ │ │ │ │ ) │
│ 693 │ │ │ │
│ ❱ 694 │ │ │ return func(*args, **kwargs) │
│ 695 │ │ │
│ 696 │ │ return wrapped │
│ 697 │
│ │
│ /home/arc-zjy3501/anaconda3/envs/sd-webui/lib/python3.10/site-packages/polygraphy/backend/trt/lo │
│ ader.py:617 in call_impl │
│ │
│ 614 │ │ """ │
│ 615 │ │ # We do not invoke super().call_impl here because we would otherwise be responsi │
│ 616 │ │ # for freeing it's return values. │
│ ❱ 617 │ │ return engine_from_bytes(super().call_impl, runtime=self._runtime) │
│ 618 │
│ 619 │
│ 620 @mod.export(funcify=True) │
│ in engine_from_bytes │
│ │
│ /home/arc-zjy3501/anaconda3/envs/sd-webui/lib/python3.10/site-packages/polygraphy/backend/base/l │
│ oader.py:40 in __call__ │
│ │
│ 37 │ │ Note: ``call_impl`` should *not* be called directly - use this function instead. │
│ 38 │ │ """ │
│ 39 │ │ __doc__ = self.call_impl.__doc__ │
│ ❱ 40 │ │ return self.call_impl(*args, **kwargs) │
│ 41 │
│ │
│ /home/arc-zjy3501/anaconda3/envs/sd-webui/lib/python3.10/site-packages/polygraphy/util/util.py:6 │
│ 94 in wrapped │
│ │
│ 691 │ │ │ │ │ │ f"Calling '{func.__qualname__}()' directly is not recommended. P │
│ 692 │ │ │ │ │ ) │
│ 693 │ │ │ │
│ ❱ 694 │ │ │ return func(*args, **kwargs) │
│ 695 │ │ │
│ 696 │ │ return wrapped │
│ 697 │
│ │
│ /home/arc-zjy3501/anaconda3/envs/sd-webui/lib/python3.10/site-packages/polygraphy/backend/trt/lo │
│ ader.py:646 in call_impl │
│ │
│ 643 │ │ Returns: │
│ 644 │ │ │ trt.ICudaEngine: The deserialized engine. │
│ 645 │ │ """ │
│ ❱ 646 │ │ buffer, owns_buffer = util.invoke_if_callable(self._serialized_engine) │
│ 647 │ │ runtime, owns_runtime = util.invoke_if_callable(self._runtime) │
│ 648 │ │ │
│ 649 │ │ trt.init_libnvinfer_plugins(trt_util.get_trt_logger(), "") │
│ │
│ /home/arc-zjy3501/anaconda3/envs/sd-webui/lib/python3.10/site-packages/polygraphy/util/util.py:6 │
│ 63 in invoke_if_callable │
│ │
│ 660 │ The second return value of this function indicates whether the argument was a callab │
│ 661 │ """ │
│ 662 │ if callable(func): │
│ ❱ 663 │ │ ret = func(*args, **kwargs) │
│ 664 │ │ return ret, True │
│ 665 │ return func, False │
│ 666 │
│ │
│ /home/arc-zjy3501/anaconda3/envs/sd-webui/lib/python3.10/site-packages/polygraphy/util/util.py:6 │
│ 94 in wrapped │
│ │
│ 691 │ │ │ │ │ │ f"Calling '{func.__qualname__}()' directly is not recommended. P │
│ 692 │ │ │ │ │ ) │
│ 693 │ │ │ │
│ ❱ 694 │ │ │ return func(*args, **kwargs) │
│ 695 │ │ │
│ 696 │ │ return wrapped │
│ 697 │
│ │
│ /home/arc-zjy3501/anaconda3/envs/sd-webui/lib/python3.10/site-packages/polygraphy/backend/trt/lo │
│ ader.py:550 in call_impl │
│ │
│ 547 │ │ │ end_time = time.time() │
│ 548 │ │ │ │
│ 549 │ │ │ if not engine_bytes: │
│ ❱ 550 │ │ │ │ G_LOGGER.critical("Invalid Engine. Please ensure the engine was built co │
│ 551 │ │ │ │
│ 552 │ │ │ G_LOGGER.finish(f"Finished engine building in {end_time - start_time:.3f} se │
│ 553 │
│ │
│ /home/arc-zjy3501/anaconda3/envs/sd-webui/lib/python3.10/site-packages/polygraphy/logger/logger. │
│ py:597 in critical │
│ │
│ 594 │ │ self.log(message, Logger.CRITICAL, stack_depth=3) │
│ 595 │ │ from polygraphy.exception import PolygraphyException │
│ 596 │ │ │
│ ❱ 597 │ │ raise PolygraphyException(message) from None │
│ 598 │ │
│ 599 │ def internal_error(self, message): │
│ 600 │ │ from polygraphy import config │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
PolygraphyException: Invalid Engine. Please ensure the engine was built correctly
Environment
TensorRT Version: 8.6.1
NVIDIA GPU: rtx4090
NVIDIA Driver Version: 525.89.02
CUDA Version: 11.6
CUDNN Version:8.4.0
Operating System: ubuntu 20.04
Python Version (if applicable): 3.10
Tensorflow Version (if applicable): none
PyTorch Version (if applicable): 1.13.1
Baremetal or Container (if so, version):
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.