[Flink] Avoid the per-consumer row round-trip when a shared native source fans out to multiple Calcs
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 21
Description
Follow-up to #2329.
When a native source feeds more than one Calc (for example `UNION ALL` over one table under Flink source reuse), the source is not fused and each Calc runs as a standalone native operator over the shared source stream. This is correct, but it repeats work per consumer.
The source emits `AuronColumnarRowData`, a zero-copy view over the native Arrow batch. Each standalone Calc then re-materializes that row back into a freshly allocated Arrow `VectorSchemaRoot`, column by column, before running its native plan (`FlinkAuronCalcOperator` -> `FlinkArrowFFIExporter` -> `FlinkArrowWriter`). So for N consumers the Kafka read and decode happen once, but the row-to-Arrow rebuild is paid N times, and it is a rows x cols scalar copy each time.
`AuronColumnarRowData` exposes only per-field getters, so there is no zero-copy re-export of the underlying batch today, and the exporter has no already-Arrow-backed fast path.
Idea: hand Arrow batches across the operator edge instead of a per-row RowData stream, so a shared native source can feed multiple native Calc operators without rebuilding the columnar buffers per consumer. This is a batch-oriented operator exchange and is larger than the multi-consumer-safety scope of #2329, hence a separate issue.
Magnitude is unmeasured. A profiler run on a `source -> single Calc` job (parallelism 1) would quantify the CPU share spent in `FlinkArrowWriter.write` / `ArrowFieldWriter` versus Kafka read and native compute, to confirm the win is worth the change.
Contributor guide
Research direction
Start with a profiler run on a parallelism-1 source-to-single-Calc job, focusing on FlinkArrowWriter.write and ArrowFieldWriter versus Kafka read and native compute. Then trace AuronColumnarRowData, FlinkAuronCalcOperator, FlinkArrowFFIExporter, and FlinkArrowWriter to understand the current row-to-Arrow path. Done means shared native sources can feed multiple native Calcs without rebuilding columnar buffers per consumer, with measurements showing the resulting trade-off.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100