matrixorigin / matrixorigin/matrixone
[Bug]: spill-disk admission rejection surfaces as 20101 for ORDER BY, GROUP BY and TOP
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
When a query reaches the configured `processLimitationSpillSize` limit,
the supported spill paths for ordinary `ORDER BY` (`MergeOrder`), high-cardinality
`GROUP BY`, and `ORDER BY ... LIMIT` (`Top`) expose an internal admission error
to MySQL clients instead of a stable resource-exhausted error.
## Environment
- Branch: `main`
- Commit: `5bc051abc44f34ccb6813e2a0bb5585d5a1db67e`
- Date: `2026-08-26`
- Deployment: fresh embedded `1 Log + 1 TN + 1 CN` cluster for every run.
- CN limits: `GuestMmuLimitation=1GiB`, `ProcessLimitationSize=1GiB`,
`ProcessLimitationSpillSize=8MiB`.
- Session: `set @@max_dop = 1`.
## Steps to reproduce
Create a wide relation so the spill-disk cap, rather than the process-memory
cap, is the limiting resource:
```sql
create table spill_probe (
id bigint not null,
k bigint not null,
payload varchar(256) not null
);
insert into spill_probe
select result,
-result,
concat(lpad(cast(result as char), 16, '0'),
repeat(char(65 + mod(result, 26)), 240))
from generate_series(1, 250000) g;
```
### MergeOrder / ordinary sort
```sql
set @@sort_spill_mem = 1;
select id, k, payload from spill_probe order by k, id;
```
### Top
```sql
set @@sort_spill_mem = 1;
select id, k, payload from spill_probe order by k, id limit 250000;
```
### High-cardinality Group
```sql
set @@agg_spill_mem = 33554432;
select id, max(payload) from group_probe group by id;
```
For the group variant, use `group_probe(id bigint not null, payload varchar(256) not null)`
and insert the same 250,000-row `id,payload` data above.
## Actual behavior
All three paths spill and then fail at exactly the configured 8 MiB cap. The
client error leaks the internal admission implementation:
```text
Error 20101 (HY000): internal error: execution resource admission rejected:
requested=65536 used=8388608 cap=8388608
```
Confirmed through the SQL protocol:
| Path | Reproduction | Result |
| --- | ---: | --- |
| `MergeOrder` (`ORDER BY`) | 3/3 | `20101/HY000` internal admission error |
| `Group` (250k distinct groups) | 1/1 | same `20101/HY000` error |
| `Top` (`ORDER BY ... LIMIT 250000`) | 1/1 | same `20101/HY000` error |
The failed queries published zero result rows. In the same connection, the
normal 10,000-row sort control immediately succeeded after every failure;
`EXPLAIN ANALYZE` for that control reported `SpillSize=`, proving the control
uses the real sort spill path. The 10,000-key aggregate control also succeeded.
## Expected behavior
The configured spill-disk cap is a supported resource limit, so an unrecoverable
admission rejection must be returned as a stable resource-exhausted error:
```text
3015 / HY000: resource exhausted: spill disk budget exceeded
```
The response must not expose `execution resource admission rejected` or use the
generic internal-error code `20101`.
## Stability and controls
- Ordinary sort reproducer: 3/3 on a fresh official-main cluster.
- The process-memory ceiling remained 1 GiB, so this is not a process-memory OOM.
- The 8 MiB cap is charged in 64 KiB writes; the rejection consistently occurred
at `used=8388608`, equal to the configured cap.
- No rows were returned before the terminal error, and the post-failure control
confirmed per-query cleanup and connection health.
## Evidence
Executed SQL-protocol probes:
```text
CGO_CFLAGS="-I$PWD/cgo -I$PWD/thirdparties/install/include" env -u GOROOT \
go test ./pkg/tests/issues/isolated \
-run '^TestCodexMergeOrderSpillCapProbe$' -count=3 -v -timeout=12m
CGO_CFLAGS="-I$PWD/cgo -I$PWD/thirdparties/install/include" env -u GOROOT \
go test ./pkg/tests/issues/isolated \
-run '^TestCodexGroupSpillCapProbe$' -count=1 -v -timeout=8m
```
The disposable probes were removed after verification; the isolated worktree
was clean on `5bc051abc44f34ccb6813e2a0bb5585d5a1db67e`.
Focused cleanup tests also pass, which distinguishes this from a leak:
```text
go test ./pkg/sql/colexec/mergeorder ./pkg/sql/colexec/group ./pkg/sql/colexec/top \
-run 'TestAccounted(MergeOrderSpillResourceAdmissionCleans|GroupSpillResourceAdmissionCleans|TopSpillResourceAdmissionCleans|MergeOrderForcedSpillKeepsOrdering|GroupForcedSpillReleasesMemoryDiskAndFD)' \
-count=1 -v -timeout=5m
```
## Code analysis
`hashbuild.TerminalBudgetError` already maps unrecoverable execution-resource
admission failures to a client-safe resource-exhausted error. The CTE-specific
fix in #27106 uses that path. In contrast, the verified `MergeOrder`, `Group`,
and `Top` call paths return spill reservation/write errors directly, allowing
the typed `ExecutionResourceError` to reach the frontend as `20101`.
`Fill` has a similar direct spill-reservation shape in static review, but it was
not included in the confirmed scope because no equivalent SQL reproducer was run.
## Regression coverage
Do not baseline the current `20101` behavior. After the fix, add:
1. SQL-protocol regression coverage for the three verified operators that
asserts `3015/HY000`, zero result rows, and a succeeding same-session control.
2. Operator unit tests that require admission errors to be normalized before
leaving `MergeOrder`, `Group`, and `Top`.
3. A threshold/spill regression at a larger scale in big-data once the
client-facing contract is fixed.
## Related
- #27106 fixed the same client-error-contract problem for materialized CTE spill.
- #26352 is not a duplicate: it concerns an unbounded `/tmp` disk-full failure
in a Big Data workflow, not an explicit `processLimitationSpillSize` rejection.
Contributor guide
Assessment
This issue has not been assessed yet.