matrixorigin / matrixorigin/matrixone
[perf] CH-benCHmark Q21/Q22 (`NOT EXISTS` → RIGHT ANTI) forced to OneCN; much slower under concurrent OLTP
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Summary
On a **3 CN** TKE MatrixOneCluster, go-tpc **CH-benCHmark** queries **Q21** and **Q22** (both contain `NOT EXISTS`, planned as **RIGHT ANTI**) always get:
```text
AP QUERY PLAN ON ONE CN(16 core)
... Table Scan ... [ForceOneCN]
```
By contrast, similar CH AP queries without ANTI (e.g. **Q5 / Q6 / Q7 / Q20**) plan as:
```text
AP QUERY PLAN ON MULTICN(16 core)
```
Q21 alone is acceptable (~4s) **still on OneCN**, but under concurrent TPC-C (-T 50) latency jumps to ~56s (~15×). Asking whether ANTI / `NOT EXISTS` must stay ForceOneCN, or can be distributed like other AP joins.
## Environment
| Item | Value |
|------|-------|
| Cluster | TKE `mo-bench-tke.yaml`, 3× CN (14c) + 1 DN + proxy |
| Image | `commit-f0ac2c9bf` (MatrixOne) |
| Namespace (repro) | `mo-chbench-commit-f0ac2c9bf-20260728` |
| Database | `tpcc_100` (100 warehouses, mo-load-data from COS) |
| Tool | pingcap/go-tpc `ch` (CH-benCHmark on TPC-C schema) |
| CI reference | https://github.com/matrixorigin/mo-nightly-regression/actions/runs/30346482771 |
### Data scale (approx.)
| Table | Rows |
|------:|-----:|
| order_line | 33,554,060 |
| stock | 10,000,000 |
| orders | 3,354,337 |
| customer | 3,000,000 |
| supplier | 10,000 |
Indexes: PRIMARY only (default TPC-C / go-tpc CH schema).
## Repro SQL (Q21, from go-tpc `ch/query.go`)
```sql
USE tpcc_100;
SELECT s_name, count(*) AS numwait
FROM supplier, order_line l1, orders, stock, nation
WHERE ol_o_id = o_id
AND ol_w_id = o_w_id
AND ol_d_id = o_d_id
AND ol_w_id = s_w_id
AND ol_i_id = s_i_id
AND mod((s_w_id * s_i_id), 10000) = s_suppkey
AND l1.ol_delivery_d > o_entry_d
AND NOT EXISTS (
SELECT *
FROM order_line l2
WHERE l2.ol_o_id = l1.ol_o_id
AND l2.ol_w_id = l1.ol_w_id
AND l2.ol_d_id = l1.ol_d_id
AND l2.ol_delivery_d > l1.ol_delivery_d
)
AND s_nationkey = n_nationkey
AND n_name = 'CHINA'
GROUP BY s_name
ORDER BY numwait DESC, s_name;
```
Q22 also uses `NOT EXISTS` and plans as **ONE CN + RIGHT ANTI + ForceOneCN**.
## Observations
### 1) Plan is ForceOneCN even when alone
```bash
EXPLAIN VERBOSE ;
```
Key fragments:
```text
AP QUERY PLAN ON ONE CN(16 core)
...
Join Type: RIGHT ANTI
Join Cond: (l2.ol_o_id = l1.ol_o_id), (l2.ol_w_id = l1.ol_w_id),
(l2.ol_d_id = l1.ol_d_id), ((l2.ol_delivery_d > l1.ol_delivery_d) IS TRUE)
-> Table Scan on tpcc_100.order_line [ForceOneCN] -- l2 side
...
-> Table Scan on tpcc_100.order_line [ForceOneCN] -- l1 side (full scan ~33.5M)
-> Table Scan on tpcc_100.orders [ForceOneCN]
-> Table Scan on tpcc_100.stock [ForceOneCN]
```
### 2) Latency: alone vs under CH OLTP
| Query | Plan | Alone wall | Under CH (-T50 -t1, CI avg) | Notes |
|-------|------|------------|------------------------------|-------|
| **Q21** | ONE CN + RIGHT ANTI | **~3.8s** | **~55.6s (~15×)** | Same ForceOneCN plan both times |
| Q22 | ONE CN + RIGHT ANTI | ~27s | ~17s | Also ForceOneCN |
| Q5 | **MULTICN** | ~17s | ~21s | No ANTI; little HTAP amplification |
| Q6 | **MULTICN** | ~0.3s | ~0.1s | Simple aggregate baseline |
| Q20 | **MULTICN** (SEMI) | ~18s | ~20s | SEMI can be multi-CN; ANTI cannot |
CH run summary (CI): tpmC ≈ 11926, **lock wait 1205 count = 0** — slowdown looks like resource/scan interference, not classic lock timeout.
### 3) Pattern
| Shape | Example | Multi-CN? |
|-------|---------|-----------|
| INNER / SEMI AP joins | Q5, Q7, Q20 | Yes (`MULTICN`) |
| `NOT EXISTS` → **RIGHT ANTI** | Q21, Q22 | No (`ONE CN` + `ForceOneCN`) |
## Ask / expected discussion
1. Is **ForceOneCN for RIGHT ANTI / NOT EXISTS** intentional (correctness / runtime limitation)?
2. If yes: can we document it, and is there a timeline for distributed ANTI?
3. If not required for correctness: please consider multi-CN ANTI (or equivalent rewrite) so CH Q21 can use 3 CNs like Q5/Q20.
4. Separately (optional follow-up): under concurrent OLTP, OneCN full scans of `order_line`/`stock`/`orders` amplify wait — any HTAP scheduling/isolation knobs worth exposing?
## Non-goals
- Not claiming wrong results.
- Not asking to add secondary indexes to TPC-C schema for benchmark fairness.
- Not comparing to stock TPC-H Q21 numbers (CH SQL is rewritten onto TPC-C tables).
## How to verify a fix
1. `EXPLAIN VERBOSE` on Q21/Q22 → prefer `AP QUERY PLAN ON MULTICN` without blanket `ForceOneCN` on large scans (or documented equivalent distributed ANTI).
2. Alone Q21 should not regress badly vs ~4s on this data size.
3. Under `go-tpc ch run -T 50 -t 1 --warehouses 100 --time 30m`, Q21 avg latency should improve materially vs ~56s on the same topology (exact target TBD).
Contributor guide
Assessment
This issue has not been assessed yet.