matrixorigin / matrixorigin/matrixone
[Bug] Window function fails on partitioned data: internal error: the Window operator currently does not support sending split result of window function
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Version
MatrixOne **v4.0.0-rc2** (`8.0.30-OmniFabric-v4.0.0-rc2`)
## Summary
A query that uses a window function with `PARTITION BY` + `ORDER BY` over a moderately large dataset (several hundred thousand rows) fails with an internal error:
```
ERROR 20101 (HY000): internal error: the Window operator currently does not support sending split result of window function.
```
The same SQL works fine on smaller inputs, so it appears the error is triggered when MO decides to split the window operator's output across nodes / pipelines.
## Reproduction
The failing query is part of a sessionization step in our Silver layer:
```sql
WITH known_markers AS (
SELECT pump, datetime, state,
MAX(CASE WHEN state != 'UNKNOWN' THEN datetime END)
OVER (PARTITION BY pump ORDER BY datetime
ROWS UNBOUNDED PRECEDING) AS last_known_dt
FROM silver_intelie_readings_classified
),
known_states AS (
SELECT DISTINCT pump, datetime AS known_dt, state AS known_state
FROM silver_intelie_readings_classified
WHERE state != 'UNKNOWN'
),
resolved AS (
SELECT km.pump, c.crew, km.datetime, km.state,
CASE WHEN km.state != 'UNKNOWN' THEN km.state
ELSE ks.known_state END AS resolved_state
FROM known_markers km
JOIN silver_intelie_readings_classified c
ON km.pump = c.pump AND km.datetime = c.datetime
LEFT JOIN known_states ks
ON km.pump = ks.pump AND km.last_known_dt = ks.known_dt
),
state_changes AS (
SELECT pump, crew, datetime, resolved_state,
CASE
WHEN LAG(resolved_state)
OVER (PARTITION BY pump ORDER BY datetime) IS NULL THEN 1
WHEN LAG(resolved_state)
OVER (PARTITION BY pump ORDER BY datetime) != resolved_state THEN 1
ELSE 0
END AS is_new_session
FROM resolved
WHERE resolved_state IS NOT NULL
),
sessions_numbered AS (
SELECT pump, crew, datetime, resolved_state,
SUM(is_new_session)
OVER (PARTITION BY pump ORDER BY datetime
ROWS UNBOUNDED PRECEDING) AS session_id
FROM state_changes
)
INSERT INTO silver_intelie_sessions
SELECT pump, crew, session_id, resolved_state,
CAST(MIN(datetime) AS TIMESTAMP(3)) AS session_start,
CAST(MAX(datetime) AS TIMESTAMP(3)) AS session_end,
COUNT(*) AS duration_minutes
FROM sessions_numbered
GROUP BY pump, crew, session_id, resolved_state;
```
Input table `silver_intelie_readings_classified` has roughly **497K rows** across 8 distinct `pump` values (~62K rows per pump). After ~11 minutes of execution the query aborts with the error above.
The same query against a smaller subset (e.g., `LIMIT 10000` in the CTE inputs) succeeds.
## Expected Behavior
Window functions should work over any size input. If MO splits the result of a window function across pipelines, the merge phase should handle it transparently.
## Actual Behavior
Window operator refuses to handle split results and aborts with an internal error message.
## Impact
- Blocks any Silver-layer sessionization pattern on production-scale data
- Standard ETL patterns (`LAG OVER PARTITION BY ...`, `SUM ... OVER ROWS UNBOUNDED PRECEDING`) are routinely used to detect state changes and number sessions — this is not a fringe SQL construct
- Our pipeline has had to fall back to limiting the date window (last 7 days only) so the window operator stays under whatever threshold triggers the split — that workaround is fragile and slows down the demo as data accumulates
## Suggested Investigation
1. Identify the threshold at which the optimizer chooses to split the window operator's output — is it row count, memory, partition count?
2. Implement merge logic on the window operator so split results can be re-assembled
3. Until the merge path lands, the optimizer should fall back to a single-partition plan (slower but correct) instead of producing this error
## Reference
- Use case: standard Medallion-architecture pipeline (Bronze sensor readings → Silver sessions → Gold KPIs)
- Window-function pattern: state-change detection via `LAG + CASE` then `SUM OVER ROWS UNBOUNDED PRECEDING` for session numbering — a textbook MySQL/PostgreSQL idiom for sessionization
- Related issues already filed: matrixone#24406, matrixone#24407, matrixone#24408
Contributor guide
Assessment
This issue has not been assessed yet.