[int8 quantization] rules for correct Q/DQ node placement with add & concat operations, unclear documentation

I want to achieve a full int8 model, with maximum speed optimizations in int8, but documentations is very unclear.

Fusion of nodes: Conv, Sigmoid, Mul, Add?

• I know that: Conv, Sigmoid, Mul will be fused
• Is it also possible to fuse Conv, Sigmoid, Mul, Add? If yes, how can I achieve this? This is unclear from doc. qdq-placement-recs

How should I handle Q/DQ nodes with concat?

• Should all the inputs/outputs be Q/DQ nodes, anything else I should know?

How should I handle Q/DQ nodes with split?

• • Should all the input/output be Q/DQ nodes, anything else I should know?

Why are the Scale & PointWise operations introduced in my graph?

1. for the two bottlenecks on the right there is some PointWise operation added for some strange reason, what is the reason for this, and why is it added?
2. Furthermore from the first conv to the second conv there is a scale introduced, what is the reason for this?
3. In the second bottleneck on the left there is a scale added, but not on the first one, what is the reason for this?
   All these operations introduces some extra latency, is the an option to omit this?

scale_pointwise

scale_pointwise_onnx

Strange behavior first "bottleneck" layer

There are extra reformat layers added in the first bottleneck layer? and why are the operations not fused? As you can see the Q/DQ nodes are correctly placed...

Environment

TensorRT Version:
8.6.3

ONNXVersion:
1.15.0, optset 17

Relevant Files

full trt graph

full onnx graph

@ttyio