matrixorigin / matrixorigin/matrixone

[Bug]: shuffle HashBuild reports no spill above join_spill_mem threshold

Open
#26,924 1 comment 0 reactions 1 assignee Claimed by @Ariznawlll View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

An eligible shuffle HashBuild remains resident, or at least reports no physical
spill, after `join_spill_mem` is set to 8,000,000 bytes. `EXPLAIN ANALYZE`
selects `shuffle: range(...)`, the Join reports 105-139 MiB of memory, and the
queries return correct results, but none of the three targeted plans contains
`SpillRows` or `SpillSize`.

This is different from #26556: the planner does select the eligible range
shuffle for both equality-predicate orders. It is also different from #26586:
the query does not fail during spill recovery and returns no budget error.

## Environment

- Branch: `main`
- MatrixOne commit: `b992c9ef36514e3d1c14760a5311eace122cce94`
- Test harness commit: `3933e729e415ec1e679301b3d8566abdd3146248`
- Date: 2026-08-10
- Deployment: isolated TKE cluster, 1 DN, 3 LogService, 3 CN and 2 Proxy
- CN: 14 CPU / 55 GiB per replica
- Session settings: `max_dop=8`, `join_spill_mem=8000000`

## Steps to reproduce

Using a source table containing at least 2,000,000 sequential integer rows:

```sql
CREATE TABLE spill_probe_2m AS
SELECT id AS k1, MOD(id, 97) AS k2, id AS payload
FROM source_100m WHERE id <= 2000000;

CREATE TABLE spill_build_2m AS
SELECT id AS k1, MOD(id, 97) AS k2, id AS payload
FROM source_100m WHERE id <= 2000000;

SET @@max_dop = 8;
SET @@join_spill_mem = 8000000;

EXPLAIN ANALYZE
SELECT COUNT(*), SUM(p.payload), SUM(b.payload)
FROM spill_probe_2m p
JOIN spill_build_2m b
ON p.k1 = b.k1 AND p.k2 = b.k2;

EXPLAIN ANALYZE
SELECT COUNT(*), SUM(p.payload), SUM(b.payload)
FROM spill_probe_2m p
JOIN spill_build_2m b
ON p.k2 = b.k2 AND p.k1 = b.k1;
```

A third variant used a 2,000,000-row build side with a 128-byte padding column:

```sql
CREATE TABLE spill_wide_build_2m AS
SELECT id AS k, CONCAT(CAST(id AS CHAR), REPEAT('x', 128)) AS pad
FROM source_100m WHERE id <= 2000000;

EXPLAIN ANALYZE
SELECT COUNT(*), SUM(p.payload), SUM(LENGTH(b.pad))
FROM spill_probe_wide_2m p
JOIN spill_wide_build_2m b ON p.k = b.k;
```

## Actual behavior

Both multi-key predicate orders choose the expected range shuffle, but neither
plan reports spill:

```text
Join MemorySize=138.45 MiB (min=16.00 MiB, max=16.00 MiB)
Join Cond: (b.k1 = p.k1), (b.k2 = p.k2) shuffle: range(b.k1)
no SpillRows / SpillSize

Join MemorySize=138.71 MiB (min=16.00 MiB, max=16.00 MiB)
Join Cond: (b.k2 = p.k2), (b.k1 = p.k1) shuffle: range(b.k1)
no SpillRows / SpillSize
```

The wide-build variant behaves the same way:

```text
Join MemorySize=105.13 MiB (min=8.00 MiB, max=38.24 MiB)
Join Cond: (b.k = p.k) shuffle: range(b.k)
no SpillRows / SpillSize
```

All queries still return the exact expected results. For the multi-key cases:

```text
joined_rows=2000000
probe_sum=2000001000000
build_sum=2000001000000
```

The post-query health control `SELECT 1` succeeds. No panic, OOM or CN restart
was observed.

## Expected behavior

For a shuffle HashBuild whose retained/build memory crosses the configured
8,000,000-byte threshold, MatrixOne should enter the supported spill path and
`EXPLAIN ANALYZE` should expose positive `SpillRows`/`SpillSize` measurements.

If the displayed Join memory is not the memory governed by
`join_spill_mem`, the plan or diagnostics should expose the resolved threshold,
unit and spill-eligibility decision so that the setting has an observable and
testable contract.

## Stability and controls

- Independent workflow reproduction: 1/1.
- Within that run: 3/3 targeted shuffle plans had no positive spill metric
(two predicate orders and one wide-build shape).
- Resident control with `join_spill_mem=0`: passed with the same exact result.
- Low-memory query results: passed and matched the resident result.
- Health/cleanup: `SELECT 1` passed; cluster remained healthy and the isolated
namespace was removed successfully.

## Evidence

- Workflow run: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/31365986312
- Failing job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/31365986312/job/93387289595
- Regression source: https://github.com/matrixorigin/mo-nightly-regression/pull/1329

The three failures are explicit plan assertions for
`regex:SpillRows=[1-9][0-9]*`; the accompanying `shuffle: range` assertions pass.

## Code analysis

The failing evidence establishes that planner shuffle selection is present and
that SQL correctness is preserved. It does not yet distinguish between:

1. the resolved `join_spill_mem` threshold not reaching the executing
HashBuild workers;
2. the threshold being interpreted per worker or in a different unit than the
session-visible value;
3. physical spill occurring but its `SpillRows`/`SpillSize` counters not being
propagated to `EXPLAIN ANALYZE`.

These are hypotheses requiring engine-side tracing; this report does not infer
one of them as the confirmed root cause.

## Regression coverage

PR matrixorigin/mo-nightly-regression#1329 contains the big-data regression.
After the product/observability contract is confirmed, retain both predicate
orders, the wide build side, resident-vs-spill result equality, positive spill
evidence and the post-recovery health check.

## Related

- matrixorigin/mo-nightly-regression#1329
- #26556: predicate-order-dependent shuffle eligibility; shuffle is present in
this reproduction.
- #26586: shared-budget spill recovery failure; no recovery error occurs here.
- #26434: spill-threshold unit and observability design.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.