matrixorigin / matrixorigin/matrixone

[Bug]: Window over large fact table under UNION ALL + outer LIMIT not short-circuited (jinpan join_adv q1004)

Open
#26,066 3 comments 0 reactions 1 assignee Claimed by @Ariznawlll View on GitHub
kind/bug severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Is there an existing issue for the same bug?

- [x] I have checked existing issues; this is about optimizer / execution efficiency (not a panic).

## Branch Name

```
main
```

## Commit ID

```
826a3e3d294ba62c7c4da217d1445f3c568ce4d9
```

Image: `registry.cn-shanghai.aliyuncs.com/matrixorigin/matrixone:nightly-826a3e3d`

## Other Environment Information

```
- Hardware: single-node MO (docker, host port 16001), rootless docker on Linux
- OS: Linux
- Dataset: jinpan_001 (dwd_s4_acdoca ~100M+ rows; filtered subset ~16.2M rows for this SQL)
- Suite: cases/jinpan_001/join_adv @q1004_adv_window_frame_running
```

## Actual Behavior

Query shape:

```text
SELECT u.* FROM (
(window query over acdoca) -- branch A
UNION ALL
(identical window query) -- branch B
) u
LIMIT 200;
```

`EXPLAIN` (abbreviated):

```text
AP QUERY PLAN ON ONE CN(64 core)
Project
Limit: 200
-> Union All
-> Window(avg) -> Partition/Sort -> Window(sum) -> Partition/Sort
-> LEFT JOIN -> Table Scan acdoca (~16.2M after filters)
-> (same full Window/Partition/Scan tree again)
```

Observed:

- Outer `LIMIT 200` sits **above** `Union All` and both `Window` operators.
- Both UNION branches are planned as full window pipelines over ~16M rows (work ≈ ×2).
- Query runs for a very long time (tens of minutes to hours+) while only needing 200 result rows.
- Filter cardinality check: `COUNT(*)` of the same `acdoca` predicates ≈ **16,231,509**.

Note: `LIMIT` cannot be pushed **below** an `UNBOUNDED PRECEDING` window without changing semantics — that part is expected. The enhancement ask is about **UNION ALL + outer LIMIT short-circuit** and avoiding full evaluation of the unused/second branch when the first branch can already satisfy LIMIT.

## Expected Behavior

For `UNION ALL` + outer `LIMIT N`:

1. Prefer evaluating branches lazily and **stop once N rows are produced** (when semantics allow).
2. Avoid scheduling a second identical full window scan when the first branch alone can supply N rows.
3. Ideally keep large window + LIMIT patterns from becoming multi-hour jobs when only a small outer LIMIT is requested (where legal).

## Steps to Reproduce

1. Load jinpan_001 (or any large fact table with similar predicates).
2. Run the SQL in `q1004.sql` (or the simplified form below).
3. Compare `EXPLAIN`: `Limit` is above `Union All` of two full `Window` trees.
4. Observe runtime far beyond what `LIMIT 200` suggests.

### Simplified shape (same plan pattern)

```sql
SELECT u.* FROM (
SELECT * FROM (
SELECT
a.rbukrs, a.belnr, a.gjahr, a.kunnr, a.wsl,
SUM(a.wsl) OVER (
PARTITION BY a.rbukrs, a.kunnr
ORDER BY a.belnr
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
) AS running_wsl,
AVG(a.wsl) OVER (
PARTITION BY a.rbukrs
ORDER BY a.belnr
ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
) AS mov_avg_3
FROM dwd_dcp.dwd_s4_acdoca a
WHERE a.drcrk = 'H'
AND a.gjahr IN ('2022','2023','2024','2025')
AND a.rbukrs IN ('1000','1100','1200','1300','2000','2100','2200','3000')
) a
UNION ALL
SELECT * FROM (
-- same SELECT as above
...
) a
) u
LIMIT 200;
```

Full SQL + EXPLAIN dump: see attached `q1004.sql` / `q1004_explain.txt` in this folder.

## Additional Context

- Case generator intended the second UNION branch to be empty (`WHERE 1 = 0`), but a wrapper bug produced two full branches; even with an empty right branch, the **left** window over ~16M rows remains expensive under outer LIMIT.
- Separate issue will cover `KILL QUERY` / `KILL CONNECTION` not canceling this long-running window query.

## Impact

Blocks jinpan `join_adv` regression progress: one SQL can occupy CN for hours even when the client only needs 200 rows.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.