microsoft / microsoft/onnxruntime

[Feature Request] Improve Xnnpack execution provider capabilities and structure by calling the Xnnpack subgraph API instead of the operator AI when creating Xnnpack layer/kernel

Open
#24,133 1 comment 0 reactions 0 assignees View on GitHub
ep:Xnnpack feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

Xnnpack execution provider currently calls the Xnnpack operatior APIs directly when creating a Xnnpack layer/kernel. For example: when creating conv layer the function "xnn_create_convolution2d_nhwc_f32" is called directly from conv_base.cc:CreateXnnpackKernel().

I gather it would be better for Xnnpack execution provider to use the Xnnpack subgraph API instead by calling the "define_" functions to create an xnnpack supported layer instead. For example: xnn_define_convolution_2d() which is part of the Xnnpack subgraph API.

Note that the Xnnpack runtime/execution capabilities does not have to be used (like Executorch and LiteRT does). Just use the Xnnpack subgraph API to handle creating of the Xnnpack layers.

By taking this approach the Xnnpack execution provider can be simplified since:
1. it does not have to handle data type mapping to correct Xnnpack kernel
2. have subgraph layer handle data validation
3. benefit from any remapping to other kernel type (this is especially useful for the 1x1 convolution kernels that can be mapped to fully connected layer... the subgraph layer would take care of this).
4. Implicit handling of reshape since appropriate reshape handling function is created during "define" call. This for instance will resolve current issues with 3Dx2D Matmul
5. automatic creation of packing layers which is needed to support ARM SME use cases
6. make the Xnnpack execution provider more futureproof as Xnnpack add new capabilities

Note that when using the subgraph API to create Xnnpack layers the actual created layer (xnn_operator type) object needs to be extracted from the subgraph object as follows: subgraph->nodes[x] ->op. (xnn_subgraph->xnn_node->xnn_operator)

### Describe scenario use case

See above

Contributor guide

Open the contributing guide

Research direction

Start in conv_base.cc at CreateXnnpackKernel() and inspect how the Xnnpack operator APIs are called directly. Trace the affected layer and kernel creation paths, then assess routing them through the Xnnpack subgraph define_ functions and extracting the created operator from the subgraph; done means the provider uses this structure while retaining the requested validation, remapping, reshape, and packing capabilities.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.