microsoft / microsoft/onnxruntime

[Performance] Quadratic complexity with SequenceMap and Scan

Open
#23,024 6 comments 0 reactions 0 assignees View on GitHub
contributions welcome performance
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

In our model we have two input tensors of same length: "id" with shape [None], sorted, and "matrices" with shape [None, 3, 3].
For each unique "id", we need to calculate the cumulative MatMul of "matrices" along axis 0 (so 3 by 3 matrix).

We achieved that by using a combination of SequenceMap and Scan. However the complexity of the ONNX graph is quadratic but we expect more linear complexity.

We observed much better performance when we broke donw the execution into batches with constant length (id_size).

### To reproduce

I ran the following code with id_size from 1k to 40k.

```python
import numpy as np
import onnxruntime
import time

# initialization
sess = onnxruntime.InferenceSession("test_perf_sequence_map_scan_graph.txt", providers=["CPUExecutionProvider"])
matrix = np.random.uniform(0, 1, 9).reshape([3, 3])
id_size = 1000
repeat_size = 100
total_size = id_size * repeat_size

# prepare test data
data = {}
data["id"] = np.repeat(np.array(range(id_size), dtype=np.int64), repeat_size)
data["matrices"] = np.repeat(np.expand_dims(matrix, 0), total_size, axis=0)
data["x"] = np.repeat(0.0, total_size)
data["y"] = np.repeat(0.0, total_size)
data["z"] = np.repeat(0.0, total_size)

# measurement
start = time.time()
res = sess.run(None, data)
end = time.time()
diff_1 = (end-start)

print("time: " + str(diff_1))
```

Results can be found here:
[perf.xlsx](https://github.com/user-attachments/files/18025531/perf.xlsx)

We observed similar results in Windows and Linux (Debian).

### Urgency

We plan to release our model in Q2 2025. So it's important and urgent for us to understand the reason behind the O(N2) complexity and if there's a way to improve. The answer could potentially change the way we use ONNX runtime.

### Platform

Windows

### OS Version

10

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.19.2

### ONNX Runtime API

Python

### Architecture

X64

### Execution Provider

Default CPU

### Execution Provider Library Version

_No response_

### Model File

[test_perf_sequence_map_scan_graph.txt](https://github.com/user-attachments/files/18025471/test_perf_sequence_map_scan_graph.txt)

### Is this a quantized model?

No

Contributor guide

Open the contributing guide

Research direction

Start by running the provided Python reproduction with test_perf_sequence_map_scan_graph.txt and varying id_size, then inspect the graph's SequenceMap and Scan operations. Compare timings across the reported input sizes and determine whether the graph explains the quadratic behavior. Done means documenting the cause and a validated path to improved performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.