microsoft / microsoft/onnxruntime
[Performance] Quadratic complexity with SequenceMap and Scan
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
In our model we have two input tensors of same length: "id" with shape [None], sorted, and "matrices" with shape [None, 3, 3].
For each unique "id", we need to calculate the cumulative MatMul of "matrices" along axis 0 (so 3 by 3 matrix).
We achieved that by using a combination of SequenceMap and Scan. However the complexity of the ONNX graph is quadratic but we expect more linear complexity.
We observed much better performance when we broke donw the execution into batches with constant length (id_size).
### To reproduce
I ran the following code with id_size from 1k to 40k.
```python
import numpy as np
import onnxruntime
import time
# initialization
sess = onnxruntime.InferenceSession("test_perf_sequence_map_scan_graph.txt", providers=["CPUExecutionProvider"])
matrix = np.random.uniform(0, 1, 9).reshape([3, 3])
id_size = 1000
repeat_size = 100
total_size = id_size * repeat_size
# prepare test data
data = {}
data["id"] = np.repeat(np.array(range(id_size), dtype=np.int64), repeat_size)
data["matrices"] = np.repeat(np.expand_dims(matrix, 0), total_size, axis=0)
data["x"] = np.repeat(0.0, total_size)
data["y"] = np.repeat(0.0, total_size)
data["z"] = np.repeat(0.0, total_size)
# measurement
start = time.time()
res = sess.run(None, data)
end = time.time()
diff_1 = (end-start)
print("time: " + str(diff_1))
```
Results can be found here:
[perf.xlsx](https://github.com/user-attachments/files/18025531/perf.xlsx)
We observed similar results in Windows and Linux (Debian).
### Urgency
We plan to release our model in Q2 2025. So it's important and urgent for us to understand the reason behind the O(N2) complexity and if there's a way to improve. The answer could potentially change the way we use ONNX runtime.
### Platform
Windows
### OS Version
10
### ONNX Runtime Installation
Released Package
### ONNX Runtime Version or Commit ID
1.19.2
### ONNX Runtime API
Python
### Architecture
X64
### Execution Provider
Default CPU
### Execution Provider Library Version
_No response_
### Model File
[test_perf_sequence_map_scan_graph.txt](https://github.com/user-attachments/files/18025471/test_perf_sequence_map_scan_graph.txt)
### Is this a quantized model?
No
Contributor guide
Research direction
Start by running the provided Python reproduction with test_perf_sequence_map_scan_graph.txt and varying id_size, then inspect the graph's SequenceMap and Scan operations. Compare timings across the reported input sizes and determine whether the graph explains the quadratic behavior. Done means documenting the cause and a validated path to improved performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100