microsoft / microsoft/onnxruntime
[Feature Request] XNNPACK EP: allow Conv/ConvTranspose with symbolic spatial dims where kernel creation doesn't depend on them
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
ConvBase::IsOnnxNodeSupported() declines any Conv/ConvTranspose whose C, H or W is unknown at graph-partitioning time.
For a graph with static channels, static H and a symbolic W, this means no convolution gets XNNPACK coverage at all, even though the kernel would be created identically for any W.
Request: allow the EP to claim Conv/ConvTranspose with symbolic spatial dims in the cases where kernel creation doesn't depend on them. C would still need to be static. If a change to the default isn't wanted, an opt-in EP option (e.g. xnnpack.allow_dynamic_spatial_dims) would leave current behaviour untouched.
Alternatives considered. We looked at exporting the model with static shapes. It's possible, but it costs us latency: with a fixed window, a single call into our streaming API needs several model invocations to drain the buffer, where a symbolic temporal axis lets one invocation cover the whole chunk. The per-invocation overhead is what we'd be paying to satisfy the shape check.
### Describe scenario use case
Streaming inference, where the temporal extent of the input is symbolic by construction. Every Conv/ConvTranspose in our graph has a static channel dim, a static H of 1, and a symbolic W, and none of them are claimed by the XNNPACK EP - Conv and ConvTranspose together account for approx 44% of node time, all of it on the CPU EP.
Repro instructions:
Environment: ONNX Runtime 1.25.0, built from source for Linux x86-64 with --use_xnnpack (C++ API; the EP registers fine and appears in GetAvailableProviders).
Step 1 — generate the model (pure onnx pip package, no ORT). One Conv, opset 17, constant initializers, explicit pads, input (1, 1, 1, "time"):
```
import numpy as np, onnx
from onnx import TensorProto, helper
rng = np.random.default_rng(0)
x = helper.make_tensor_value_info("X", TensorProto.FLOAT, [1, 1, 1, "time"])
y = helper.make_tensor_value_info("Y", TensorProto.FLOAT, [1, 8, 1, "time_out"])
w = helper.make_tensor("W", TensorProto.FLOAT, (8, 1, 1, 3), rng.standard_normal((8, 1, 1, 3)).astype(np.float32).flatten())
b = helper.make_tensor("B", TensorProto.FLOAT, (8,), rng.standard_normal(8).astype(np.float32))
conv = helper.make_node("Conv", ["X", "W", "B"], ["Y"], name="conv1",
kernel_shape=[1, 3], strides=[1, 1], pads=[0, 1, 0, 1], dilations=[1, 1], group=1)
graph = helper.make_graph([conv], "g", [x], [y], initializer=[w, b])
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)], ir_version=8)
onnx.checker.check_model(model)
onnx.save(model, "conv_min.onnx")
```
Step 2 — repro program (repro_min.cpp). Two sessions from the same file; the free-dimension override is the only difference:
```
#include
#include
#include
int main() {
Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "repro");
for (bool override_dim : {false, true}) {
Ort::SessionOptions opts;
opts.EnableProfiling(override_dim ? "prof_fixed" : "prof_symbolic");
if (override_dim) opts.AddFreeDimensionOverrideByName("time", 512);
opts.AppendExecutionProvider("XNNPACK", {{"intra_op_num_threads", "1"}});
Ort::Session session(env, "conv_min.onnx", opts);
std::vector input(512, 0.5f);
std::vector shape{1, 1, 1, 512};
auto mem = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
Ort::Value x = Ort::Value::CreateTensor(mem, input.data(), input.size(), shape.data(), shape.size());
const char* in[] = {"X"}; const char* out[] = {"Y"};
session.Run(Ort::RunOptions{nullptr}, in, &x, 1, out, 1);
Ort::AllocatorWithDefaultOptions alloc;
std::cout << (override_dim ? "fixed " : "symbolic") << " profile: "
<< session.EndProfilingAllocated(alloc).get() << "\n";
}
}
```
Step 3 — build and run:
```
g++ -std=c++17 -O2 repro_min.cpp -I /include -L /lib -lonnxruntime -Wl,-rpath,/lib -o repro_min
./repro_min
jq -r '.[] | select(.cat=="Node" and .args.op_name=="Conv") | [.name, .args.provider] | @tsv' prof_*.json
```
Observed:
```
== prof_symbolic_*.json ==
Y_nchwc_kernel_time CPUExecutionProvider
== prof_fixed_*.json ==
conv1_token_1_kernel_time XnnpackExecutionProvider
```
Expected: XNNPACK to claim the Conv in both cases (or documentation stating that the XNNPACK EP requires static spatial dims at partition time), since the node is identical and the actual run shape is static.
Contributor guide
Research direction
Start at ConvBase::IsOnnxNodeSupported() in the XNNPACK execution-provider code and compare the symbolic-dimension path with the fixed-dimension path described by repro_min.cpp. Run the supplied model and profiling commands first, then verify that Conv and ConvTranspose with static channels and eligible symbolic spatial dimensions are claimed without changing cases where kernel creation depends on shape.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100