matrixorigin / matrixorigin/matrixone
[Bug]: shuffle HashBuild reports no spill above join_spill_mem threshold
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
An eligible shuffle HashBuild remains resident, or at least reports no physical
spill, after `join_spill_mem` is set to 8,000,000 bytes. `EXPLAIN ANALYZE`
selects `shuffle: range(...)`, the Join reports 105-139 MiB of memory, and the
queries return correct results, but none of the three targeted plans contains
`SpillRows` or `SpillSize`.
This is different from #26556: the planner does select the eligible range
shuffle for both equality-predicate orders. It is also different from #26586:
the query does not fail during spill recovery and returns no budget error.
## Environment
- Branch: `main`
- MatrixOne commit: `b992c9ef36514e3d1c14760a5311eace122cce94`
- Test harness commit: `3933e729e415ec1e679301b3d8566abdd3146248`
- Date: 2026-08-10
- Deployment: isolated TKE cluster, 1 DN, 3 LogService, 3 CN and 2 Proxy
- CN: 14 CPU / 55 GiB per replica
- Session settings: `max_dop=8`, `join_spill_mem=8000000`
## Steps to reproduce
Using a source table containing at least 2,000,000 sequential integer rows:
```sql
CREATE TABLE spill_probe_2m AS
SELECT id AS k1, MOD(id, 97) AS k2, id AS payload
FROM source_100m WHERE id <= 2000000;
CREATE TABLE spill_build_2m AS
SELECT id AS k1, MOD(id, 97) AS k2, id AS payload
FROM source_100m WHERE id <= 2000000;
SET @@max_dop = 8;
SET @@join_spill_mem = 8000000;
EXPLAIN ANALYZE
SELECT COUNT(*), SUM(p.payload), SUM(b.payload)
FROM spill_probe_2m p
JOIN spill_build_2m b
ON p.k1 = b.k1 AND p.k2 = b.k2;
EXPLAIN ANALYZE
SELECT COUNT(*), SUM(p.payload), SUM(b.payload)
FROM spill_probe_2m p
JOIN spill_build_2m b
ON p.k2 = b.k2 AND p.k1 = b.k1;
```
A third variant used a 2,000,000-row build side with a 128-byte padding column:
```sql
CREATE TABLE spill_wide_build_2m AS
SELECT id AS k, CONCAT(CAST(id AS CHAR), REPEAT('x', 128)) AS pad
FROM source_100m WHERE id <= 2000000;
EXPLAIN ANALYZE
SELECT COUNT(*), SUM(p.payload), SUM(LENGTH(b.pad))
FROM spill_probe_wide_2m p
JOIN spill_wide_build_2m b ON p.k = b.k;
```
## Actual behavior
Both multi-key predicate orders choose the expected range shuffle, but neither
plan reports spill:
```text
Join MemorySize=138.45 MiB (min=16.00 MiB, max=16.00 MiB)
Join Cond: (b.k1 = p.k1), (b.k2 = p.k2) shuffle: range(b.k1)
no SpillRows / SpillSize
Join MemorySize=138.71 MiB (min=16.00 MiB, max=16.00 MiB)
Join Cond: (b.k2 = p.k2), (b.k1 = p.k1) shuffle: range(b.k1)
no SpillRows / SpillSize
```
The wide-build variant behaves the same way:
```text
Join MemorySize=105.13 MiB (min=8.00 MiB, max=38.24 MiB)
Join Cond: (b.k = p.k) shuffle: range(b.k)
no SpillRows / SpillSize
```
All queries still return the exact expected results. For the multi-key cases:
```text
joined_rows=2000000
probe_sum=2000001000000
build_sum=2000001000000
```
The post-query health control `SELECT 1` succeeds. No panic, OOM or CN restart
was observed.
## Expected behavior
For a shuffle HashBuild whose retained/build memory crosses the configured
8,000,000-byte threshold, MatrixOne should enter the supported spill path and
`EXPLAIN ANALYZE` should expose positive `SpillRows`/`SpillSize` measurements.
If the displayed Join memory is not the memory governed by
`join_spill_mem`, the plan or diagnostics should expose the resolved threshold,
unit and spill-eligibility decision so that the setting has an observable and
testable contract.
## Stability and controls
- Independent workflow reproduction: 1/1.
- Within that run: 3/3 targeted shuffle plans had no positive spill metric
(two predicate orders and one wide-build shape).
- Resident control with `join_spill_mem=0`: passed with the same exact result.
- Low-memory query results: passed and matched the resident result.
- Health/cleanup: `SELECT 1` passed; cluster remained healthy and the isolated
namespace was removed successfully.
## Evidence
- Workflow run: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/31365986312
- Failing job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/31365986312/job/93387289595
- Regression source: https://github.com/matrixorigin/mo-nightly-regression/pull/1329
The three failures are explicit plan assertions for
`regex:SpillRows=[1-9][0-9]*`; the accompanying `shuffle: range` assertions pass.
## Code analysis
The failing evidence establishes that planner shuffle selection is present and
that SQL correctness is preserved. It does not yet distinguish between:
1. the resolved `join_spill_mem` threshold not reaching the executing
HashBuild workers;
2. the threshold being interpreted per worker or in a different unit than the
session-visible value;
3. physical spill occurring but its `SpillRows`/`SpillSize` counters not being
propagated to `EXPLAIN ANALYZE`.
These are hypotheses requiring engine-side tracing; this report does not infer
one of them as the confirmed root cause.
## Regression coverage
PR matrixorigin/mo-nightly-regression#1329 contains the big-data regression.
After the product/observability contract is confirmed, retain both predicate
orders, the wide build side, resident-vs-spill result equality, positive spill
evidence and the post-recovery health check.
## Related
- matrixorigin/mo-nightly-regression#1329
- #26556: predicate-order-dependent shuffle eligibility; shuffle is present in
this reproduction.
- #26586: shared-budget spill recovery failure; no recovery error occurs here.
- #26434: spill-threshold unit and observability design.
Contributor guide
Assessment
This issue has not been assessed yet.