matrixorigin / matrixorigin/matrixone

[Performance]: Complete special-operator pruning and use low-cost row carriers

Open
#25,612 0 comments 0 reactions 1 assignee Claimed by @Ariznawlll View on GitHub
kind/performance severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Is there an existing issue for performance?

- [x] I have checked the existing issues.

## Environment

- Version or commit-id: main at `7821d21c72`
- Hardware parameters: N/A
- OS type: Linux
- Others: logical-plan column pruning in `QueryBuilder.remapAllColRefs`

## Details of Performance

After closing the general `UNION ALL`, `AGG`, and unused-window gaps, several special operators and row-carrier fallbacks still retain avoidable work:

1. `SAMPLE` marks every sample expression as required even if only a subset of sampled outputs is consumed.
2. `TIME_WINDOW` retains every aggregate expression and input even when an outer query discards some aggregate outputs.
3. Zero-column fallbacks choose position/column/aggregate 0 for table/function scans, `UNION ALL`, and scalar aggregates. The first candidate can be a wide varlen column or an expensive aggregate such as `group_concat`, causing avoidable I/O, allocation, and CPU work.

The target is to minimize physical input width and operator state without changing row-count semantics. In particular, `SAMPLE` and scalar aggregation may still need a row-producing carrier when all values are discarded.

## Additional information

Proposed design constraints:

- Model row cardinality as a first-class requirement rather than implicitly retaining slot 0.
- Prefer zero-column batches if all affected execution operators support them; otherwise choose a deterministic low-cost carrier using type width and aggregate cost.
- Compact `SAMPLE`/`TIME_WINDOW` physical slots and remap `HAVING`, special `_wstart`/`_wend` outputs, merge-stage state, and parent projections together.
- Preserve side-effecting expressions and all group/order/partition dependencies.
- Add plan-shape tests, empty-input and 0/1/many-row result tests, wide-varlen performance harnesses, and allocation/peak-memory measurements.

Related architectural work: #14077.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.