flagos-ai / flagos-ai/FlagPerf

stable_diffusion_v1_4使用torchtrt推理时报错

Open
#754 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
379
Forks
124
PR merge metrics
No merged PRs in 30d

Description

ERROR: [Torch-TensorRT] - Unsupported operator: aten::to.dtype_layout(Tensor(a) self, *, ScalarType? dtype=None, Layout? layout=None, Device? device=None, bool? pin_memory=None, bool non_blocking=False, bool copy=False, MemoryFormat? memory_format=None) -> Tensor(a)
/workspace/luor/FlagPerf/inference/benchmarks/stable_diffusion_v1_4/pytorch/model_utils/unet2d.py(854): forward
/root/miniconda3/envs/fperf/lib/python3.10/site-packages/torch/nn/modules/module.py(1501): _slow_forward
/root/miniconda3/envs/fperf/lib/python3.10/site-packages/torch/nn/modules/module.py(1520): _call_impl
/root/miniconda3/envs/fperf/lib/python3.10/site-packages/torch/nn/modules/module.py(1511): _wrapped_call_impl
/root/miniconda3/envs/fperf/lib/python3.10/site-packages/torch/jit/_trace.py(1074): trace_module
/root/miniconda3/envs/fperf/lib/python3.10/site-packages/torch/jit/_trace.py(806): trace
/workspace/luor/FlagPerf/inference/inference_engine/nvidia/torchtrt.py(23): __call__
/workspace/luor/FlagPerf/inference/benchmarks/stable_diffusion_v1_4/pytorch/forward.py(209): engine_forward
/workspace/luor/FlagPerf/inference/run_inference.py(80): main
/workspace/luor/FlagPerf/inference/run_inference.py(137):

Traceback (most recent call last):
File "/workspace/luor/FlagPerf/inference/run_inference.py", line 137, in
config, p_forward, p_infer, p_forward_core, p_infer_core, val_acc, infer_acc = main(
File "/workspace/luor/FlagPerf/inference/run_inference.py", line 80, in main
p_infer, p_infer_core, infer_acc = benchmark_module.engine_forward(
File "/workspace/luor/FlagPerf/inference/benchmarks/stable_diffusion_v1_4/pytorch/forward.py", line 209, in engine_forward
outputs = model(inputs)
File "/workspace/luor/FlagPerf/inference/inference_engine/nvidia/torchtrt.py", line 25, in __call__
self.trt_model = torchtrt.compile(
File "/root/miniconda3/envs/fperf/lib/python3.10/site-packages/torch_tensorrt/_compile.py", line 185, in compile
compiled_ts_module: torch.jit.ScriptModule = torchscript_compile(
File "/root/miniconda3/envs/fperf/lib/python3.10/site-packages/torch_tensorrt/ts/_compiler.py", line 151, in compile
compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: [Error thrown at core/compiler.cpp:413] Expected conversion::VerifyConverterSupportForBlock(g->block()) to be true but got false
Not all operations in graph are supported by the compiler

请问测试时,官方使用的软件版本是多少?
以下是我测试时的版本:
libtorch 2.4.0
torch 2.2.2
torch-tensorrt 2.2.0
torchmetrics 1.4.2
tensorrt 8.6.1.post1
tensorrt-bindings 8.6.1
tensorrt-libs 8.6.1

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with inference/benchmarks/stable_diffusion_v1_4/pytorch/model_utils/unet2d.py at line 854 and trace the call through inference/inference_engine/nvidia/torchtrt.py line 23. Reproduce the failure with the listed PyTorch, Torch-TensorRT, and TensorRT versions, then compare the supported software versions and verify that stable-diffusion inference compiles without the unsupported aten::to.dtype_layout operator.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.