matrixorigin / matrixorigin/matrixone
[Bug]: TPC-DS 1TB Q80 join-spill cannot admit minimum allocation at 40 GiB
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
TPC-DS 1 TB `query80` fails on single-CN MatrixOne main when join-spill reaches the 40 GiB process memory ceiling. The spill path cannot admit even its minimum scatter/codec allocation, so the supported query aborts instead of continuing through spill.
## Environment
- Branch: `main`
- Commit: `e31ee06042bc708ac5620a579715e07ec31c21b6`
- Deployment: 129 single node / one CN, TPC-DS SF=1000 (1 TB)
- Query process limitation: `42949672960` bytes (40 GiB)
- Run: https://github.com/matrixorigin/mo-auto-test/actions/runs/30991050343
- Job: https://github.com/matrixorigin/mo-auto-test/actions/runs/30991050343/job/92258474435
- Query window: 2026-08-05 17:18:38–17:20:32 CST
## Steps to reproduce
Load TPC-DS SF=1000 and execute the exact generated SQL:
```sql
with ssr as
(select s_store_id as store_id,
sum(ss_ext_sales_price) as sales,
sum(coalesce(sr_return_amt, 0)) as returns,
sum(ss_net_profit - coalesce(sr_net_loss, 0)) as profit
from store_sales left outer join store_returns on
(ss_item_sk = sr_item_sk and ss_ticket_number = sr_ticket_number),
date_dim,
store,
item,
promotion
where ss_sold_date_sk = d_date_sk
and d_date between cast('1998-08-04' as date)
and (cast('1998-08-04' as date) + 30 days)
and ss_store_sk = s_store_sk
and ss_item_sk = i_item_sk
and i_current_price > 50
and ss_promo_sk = p_promo_sk
and p_channel_tv = 'N'
group by s_store_id)
,
csr as
(select cp_catalog_page_id as catalog_page_id,
sum(cs_ext_sales_price) as sales,
sum(coalesce(cr_return_amount, 0)) as returns,
sum(cs_net_profit - coalesce(cr_net_loss, 0)) as profit
from catalog_sales left outer join catalog_returns on
(cs_item_sk = cr_item_sk and cs_order_number = cr_order_number),
date_dim,
catalog_page,
item,
promotion
where cs_sold_date_sk = d_date_sk
and d_date between cast('1998-08-04' as date)
and (cast('1998-08-04' as date) + 30 days)
and cs_catalog_page_sk = cp_catalog_page_sk
and cs_item_sk = i_item_sk
and i_current_price > 50
and cs_promo_sk = p_promo_sk
and p_channel_tv = 'N'
group by cp_catalog_page_id)
,
wsr as
(select web_site_id,
sum(ws_ext_sales_price) as sales,
sum(coalesce(wr_return_amt, 0)) as returns,
sum(ws_net_profit - coalesce(wr_net_loss, 0)) as profit
from web_sales left outer join web_returns on
(ws_item_sk = wr_item_sk and ws_order_number = wr_order_number),
date_dim,
web_site,
item,
promotion
where ws_sold_date_sk = d_date_sk
and d_date between cast('1998-08-04' as date)
and (cast('1998-08-04' as date) + 30 days)
and ws_web_site_sk = web_site_sk
and ws_item_sk = i_item_sk
and i_current_price > 50
and ws_promo_sk = p_promo_sk
and p_channel_tv = 'N'
group by web_site_id)
select * from ( select channel
, id
, sum(sales) as sales
, sum(returns) as returns
, sum(profit) as profit
from
(select 'store channel' as channel
, 'store' || store_id as id
, sales
, returns
, profit
from ssr
union all
select 'catalog channel' as channel
, 'catalog_page' || catalog_page_id as id
, sales
, returns
, profit
from csr
union all
select 'web channel' as channel
, 'web_site' || web_site_id as id
, sales
, returns
, profit
from wsr
) x
group by rollup (channel, id)
order by channel
,id
) where rownum <= 100;
```
## Actual behavior
The query failed after 114.623 seconds:
```text
ERROR 1105 (HY000) at line 1:
minimum allocation cannot be admitted:
owner=join-spill
site=scatter-selected-or-codec
used=42872692992
limit=42949672960
```
Server evidence:
- pipeline MPool reached `42949737400` bytes, approximately 40.0 GiB;
- 169 `minimum allocation cannot be admitted` records were logged;
- the failing owner/site was the join-spill scatter/codec path;
- 27 follow-on `context canceled` records were logged;
- MatrixOne did not panic or restart.
## Expected behavior
The join-spill path should reserve enough bounded recovery/scatter capacity to spill admitted state and complete the supported TPC-DS query while honoring the hard process limit. It should not consume the full ceiling and then fail its own minimum recovery allocation.
## Stability and controls
- Reproducer: observed in one official full TPC-DS SF=1000 run; no manual rerun before filing.
- Control: 84 of 99 queries completed successfully in the same run.
- Failure state: MatrixOne stayed alive and later queries continued.
## Evidence
- Actions run/job linked above.
- Exact generated SQL and `query80.err` are in the run's `tpcds-result` artifact.
- Loki logs for host `10-222-1-129` show the allocation-admission series and MPool peak.
## Code analysis
This is a join-spill allocation/reservation failure, distinct from the HashBuild budget error reported for query4/query64. The spill path reaches the process limit before reserving the minimum scatter-selected/codec allocation needed to make forward progress. The follow-on context cancellations are consequences, not the initiating error.
The workflow's intended `GOMEMLIMIT` was placed after command redirection and was not exported to `mo-service`; that infrastructure problem will be fixed separately. The error here is nevertheless MatrixOne's explicit query process-limit/spill-recovery failure at 40 GiB.
## Regression coverage
Add large-data join-spill coverage for the query80 plan shape under a finite process limit. Assert query completion, result correctness, bounded peak memory, spill cleanup, and successful execution of a following statement.
## Related
- Source run: https://github.com/matrixorigin/mo-auto-test/actions/runs/30991050343
- HashBuild shared-budget issue (different owner/path): #26586
Contributor guide
Assessment
This issue has not been assessed yet.