matrixorigin / matrixorigin/matrixone
[Bug]: Window over large fact table under UNION ALL + outer LIMIT not short-circuited (jinpan join_adv q1004)
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Is there an existing issue for the same bug?
- [x] I have checked existing issues; this is about optimizer / execution efficiency (not a panic).
## Branch Name
```
main
```
## Commit ID
```
826a3e3d294ba62c7c4da217d1445f3c568ce4d9
```
Image: `registry.cn-shanghai.aliyuncs.com/matrixorigin/matrixone:nightly-826a3e3d`
## Other Environment Information
```
- Hardware: single-node MO (docker, host port 16001), rootless docker on Linux
- OS: Linux
- Dataset: jinpan_001 (dwd_s4_acdoca ~100M+ rows; filtered subset ~16.2M rows for this SQL)
- Suite: cases/jinpan_001/join_adv @q1004_adv_window_frame_running
```
## Actual Behavior
Query shape:
```text
SELECT u.* FROM (
(window query over acdoca) -- branch A
UNION ALL
(identical window query) -- branch B
) u
LIMIT 200;
```
`EXPLAIN` (abbreviated):
```text
AP QUERY PLAN ON ONE CN(64 core)
Project
Limit: 200
-> Union All
-> Window(avg) -> Partition/Sort -> Window(sum) -> Partition/Sort
-> LEFT JOIN -> Table Scan acdoca (~16.2M after filters)
-> (same full Window/Partition/Scan tree again)
```
Observed:
- Outer `LIMIT 200` sits **above** `Union All` and both `Window` operators.
- Both UNION branches are planned as full window pipelines over ~16M rows (work ≈ ×2).
- Query runs for a very long time (tens of minutes to hours+) while only needing 200 result rows.
- Filter cardinality check: `COUNT(*)` of the same `acdoca` predicates ≈ **16,231,509**.
Note: `LIMIT` cannot be pushed **below** an `UNBOUNDED PRECEDING` window without changing semantics — that part is expected. The enhancement ask is about **UNION ALL + outer LIMIT short-circuit** and avoiding full evaluation of the unused/second branch when the first branch can already satisfy LIMIT.
## Expected Behavior
For `UNION ALL` + outer `LIMIT N`:
1. Prefer evaluating branches lazily and **stop once N rows are produced** (when semantics allow).
2. Avoid scheduling a second identical full window scan when the first branch alone can supply N rows.
3. Ideally keep large window + LIMIT patterns from becoming multi-hour jobs when only a small outer LIMIT is requested (where legal).
## Steps to Reproduce
1. Load jinpan_001 (or any large fact table with similar predicates).
2. Run the SQL in `q1004.sql` (or the simplified form below).
3. Compare `EXPLAIN`: `Limit` is above `Union All` of two full `Window` trees.
4. Observe runtime far beyond what `LIMIT 200` suggests.
### Simplified shape (same plan pattern)
```sql
SELECT u.* FROM (
SELECT * FROM (
SELECT
a.rbukrs, a.belnr, a.gjahr, a.kunnr, a.wsl,
SUM(a.wsl) OVER (
PARTITION BY a.rbukrs, a.kunnr
ORDER BY a.belnr
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
) AS running_wsl,
AVG(a.wsl) OVER (
PARTITION BY a.rbukrs
ORDER BY a.belnr
ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
) AS mov_avg_3
FROM dwd_dcp.dwd_s4_acdoca a
WHERE a.drcrk = 'H'
AND a.gjahr IN ('2022','2023','2024','2025')
AND a.rbukrs IN ('1000','1100','1200','1300','2000','2100','2200','3000')
) a
UNION ALL
SELECT * FROM (
-- same SELECT as above
...
) a
) u
LIMIT 200;
```
Full SQL + EXPLAIN dump: see attached `q1004.sql` / `q1004_explain.txt` in this folder.
## Additional Context
- Case generator intended the second UNION branch to be empty (`WHERE 1 = 0`), but a wrapper bug produced two full branches; even with an empty right branch, the **left** window over ~16M rows remains expensive under outer LIMIT.
- Separate issue will cover `KILL QUERY` / `KILL CONNECTION` not canceling this long-running window query.
## Impact
Blocks jinpan `join_adv` regression progress: one SQL can occupy CN for hours even when the client only needs 200 rows.
Contributor guide
Assessment
This issue has not been assessed yet.