matrixorigin / matrixorigin/matrixone
[Feature Request]: Substrait plan offload to Sirius sidecar
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Summary
Introduce a production-grade MatrixOne-to-Sirius sidecar path in which MatrixOne exports an eligible optimized logical SQL query as Substrait, Sirius executes it through its GPU planner, and results stream back under the original MySQL query's memory, cancellation, authentication, and snapshot-visibility guarantees.
This replaces the prototype SQL-over-HTTP and fully buffered JSON path, not merely its serialization format.
## Design boundaries
- Export a complete bounded logical query, never a MatrixOne distributed physical pipeline and never a phase-one hybrid plan.
- Represent TAE input as a versioned `ReadRel.ExtensionTable` with an opaque, authenticated read reference. Plans contain no manifest URLs, object-store credentials, or reusable capabilities.
- Pin snapshot, schema, immutable object set, and retention lease before execution. Phase one rejects unflushed data and visibility-dependent tombstones.
- Import Substrait into a DuckDB logical plan but execute it through the Sirius physical planner, never ordinary DuckDB CPU execution.
- Stream Arrow Flight batches through MatrixOne's existing MySQL result writer with bounded in-flight memory and exactly-once cancellation and cleanup.
- Fall back to native MatrixOne only before any result schema or row is visible.
## Delivered foundations
- matrixorigin/matrixone#26660: reusable differential runner, typed result comparison, execution evidence, and redacted failure artifacts.
- matrixorigin/sirius#3: strict `TaeRead` Substrait execution boundary and direct Sirius planning.
- matrixorigin/mo-sirius-sidecar#5: authenticated Arrow Flight runtime, resolver integration, bounded streaming, and sidecar cancellation.
- matrixorigin/matrixone#26737: bounded MatrixOne Substrait exporter, snapshot-safe TAE admission, durable resolver leases, GC protection, and compiler cut point.
## TPC-H implementation under review
The remaining product implementation is filed as four dependency-ordered PRs:
1. matrixorigin/duckdb-substrait#2 — import MatrixOne left/right anti joins with SQL NULL and duplicate semantics.
2. matrixorigin/sirius#4 — validate the exact TPC-H relation/expression/type/function surface, seven required join modes, authenticated `database_id`, and bounded backward references.
3. matrixorigin/mo-sirius-sidecar#6 — publish the exact capability contract and bind Flight ticket, cancellation, quiescence, resolver, and lease ownership.
4. matrixorigin/matrixone#26981 — export every canonical optimized TPC-H Q1-Q22 plan and integrate exact Flight streaming into CN/frontend/compiler lifecycle.
MatrixOne's Q15 CTE is already inlined in the canonical optimized plan. None of these PRs changes CTE planning. Bounded `Sink`/`ReferenceRel` support is contract coverage only, not a prerequisite for the canonical Q15 export.
## Remaining after the four PRs merge
#26159 is the only substantive remaining delivery:
- run all 22 TPC-H queries natively and offloaded and compare typed results;
- prove actual Sirius GPU execution rather than DuckDB CPU fallback; and
- exercise storage races, slow clients, disconnects, cancellation, sidecar failure, schema mismatch, and cleanup.
## Subtasks
- #26155 — Sirius/importer/sidecar TPC-H capability stack: implementation filed in the first three PRs above.
- #26156 — MatrixOne TPC-H exporter: implementation filed in matrixorigin/matrixone#26981.
- #26157 — snapshot-correct read admission and durable leases: complete.
- #26158 — Flight-to-MySQL runtime integration: implementation filed in matrixorigin/mo-sirius-sidecar#6 and matrixorigin/matrixone#26981.
- #26159 — all-22-query differential, GPU, storage-race, and lifecycle E2E proof: remains.
## Merge order
**duckdb-substrait#2 -> sirius#4 -> mo-sirius-sidecar#6 -> matrixone#26981 -> #26159 E2E closure.**
Full mutable-table support with snapshot-correct tombstone application remains outside phase one. Unsafe states must fall back before execution, never return approximate results.
Contributor guide
Assessment
This issue has not been assessed yet.