matrixorigin / matrixorigin/matrixone
[Feature Request]: Embed Sirius in MatrixOne through CGo and retire Flight offload
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Is there an existing issue for the same feature request?
- [x] I have checked the existing issues.
## Is your feature request related to a problem?
The existing Sirius integration runs a separate Flight sidecar. Its transport,
batch framing, duplicate buffering, deployment and recovery lifecycle are not
needed when MO and Sirius run in the same process. Upstream's existing C++ FFI
is not a bounded replacement: it materializes results before exposing Arrow,
and its generic streaming repositories have no channel back-pressure.
Predecessors: #26154, #27586 and PR #27599. Reuse the evidence and regression
coverage in #26159 without redefining those issues as the embedding project.
## Describe the feature you'd like
Statically link Sirius and DuckDB into mo-service through a C ABI/CGo bridge,
with pinned shared GPU dependencies. Start from matrixorigin/sirius's
upstream-dev-merge synchronized to sirius-db/sirius:dev.
- Default to MO readers; preserve explicit direct TAE input and its admission restrictions.
- Batch on demand with bounded prefetch; native back-pressure must stop MO readers.
- Bound input AND incremental output; no whole-table transport/result buffering.
- One active query on the selected GPU initially, bounded admission, GPU streams=2 by default.
- Preserve CPU-only builds and ordinary native MO execution.
- Retain Flight until all 22 SF1/SF10 queries actually offload through both embedded scan modes and lifecycle/performance gates pass.
- Track exact wide-decimal compatibility separately; CPU fallback does not satisfy parity.
- Fatal native/CUDA failures may require restarting mo-service, not just a sidecar.
## Describe implementation you've considered
Approved implementation plan: ten migration PRs (five Sirius, four MO, one sidecar).
The implementation series will carry docs/design/sirius-embedded.md with the
complete API, ownership, capacity, lifecycle, validation and rollout contracts.
| PR | Repository | Delivery | Dependencies |
| --- | --- | --- | --- |
| 1 | Sirius | Synchronize upstream-dev-merge with newest upstream dev | none |
| 2 | MO | Versioned design, narrow backend contract and configuration | approved design |
| 3 | Sirius | Static embedding target, C ABI and query lifecycle | 1, contract from 2 |
| 4 | Sirius | Bounded MO-native ingestion | 3 |
| 5 | Sirius | Embedded TAE bindings and strict Substrait admission | 3, 4 |
| 6 | Sirius | Incremental bounded native result sink | 3; integration with 4, 5 |
| 7 | MO | CGo bridge, build/package and service ownership | 2, 3-6 |
| 8 | MO | Real scans/results and parity harness, opt-in backend | 7 |
| 9 | MO | Embedded cutover and Flight removal | 8, numeric compatibility, all gates |
| 10 | sidecar | Retire MO/Sirius Flight deployment and stale code/tests | 9, release artifact |
- [x] PR 1: upstream synchronization — https://github.com/matrixorigin/sirius/pull/13 (implemented and locally validated; awaiting merge)
- [ ] PR 2: MO contract/configuration — #28973 (implemented and locally validated; exact-head CI running)
- [ ] PR 3: native ABI/lifecycle
- [ ] PR 4: bounded MO input
- [ ] PR 5: TAE/Substrait binding
- [ ] PR 6: incremental results
- [ ] PR 7: MO CGo/build
- [ ] PR 8: real integration/parity harness
- [ ] Separate numeric compatibility blocker #28968 resolved
- [ ] All-22 correctness, actual GPU routing, bounded memory, cancellation and performance gates pass
- [ ] PR 9: MO cutover/removal
- [ ] PR 10: sidecar cleanup/release instructions
## Documentation, Adoption, Use Case, Migration Strategy
Use existing working directories and dedicated implementation branches. Do not
expand the old #27599 / sirius#12 / mo-sirius-sidecar#20 PRs into this migration.
Keep their evidence and reusable tests; supersede them explicitly only after
replacement behavior is delivered. No predecessor closure is authorized by
creating this issue.
Sirius integration PRs target matrixorigin/sirius:upstream-dev-merge; MO and
sidecar PRs target main. Pin native consumers to merged Sirius commits.
All Sirius builds/tests use Pixi and incremental host builds, not container rebuilds.
Intermediate PRs reference this issue; they must not auto-close it.
## Additional information
Acceptance: all 22 SF1/SF10 queries through embedded MO and TAE with two GPU
streams, no CPU fallback, correct schema/results and bounded cleanup. Validate
streams=1 and a higher-stream stress configuration too. Publish all 22 client
wall times and sums for MO native, Flight+TAE, Flight+MO, embedded+TAE and
embedded+MO, with exact revisions and raw repeated runs. Preserve the 2x
MO-reader/direct-TAE gates for Q9 and the suite median, and require no regression
against matching controlled Flight baselines. Historical 34.16s/15.50s results
are not fresh acceptance evidence.
Milestones: PRs 1-8 deliver a disabled-by-default embedded backend; resolving
the numeric blocker and passing every gate allows PRs 9-10 and migration closure.
Contributor guide
Assessment
This issue has not been assessed yet.