apache / apache/auron

[Flink] Avoid the per-consumer row round-trip when a shared native source fans out to multiple Calcs

Open
#2,416 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

Follow-up to #2329.

When a native source feeds more than one Calc (for example `UNION ALL` over one table under Flink source reuse), the source is not fused and each Calc runs as a standalone native operator over the shared source stream. This is correct, but it repeats work per consumer.

The source emits `AuronColumnarRowData`, a zero-copy view over the native Arrow batch. Each standalone Calc then re-materializes that row back into a freshly allocated Arrow `VectorSchemaRoot`, column by column, before running its native plan (`FlinkAuronCalcOperator` -> `FlinkArrowFFIExporter` -> `FlinkArrowWriter`). So for N consumers the Kafka read and decode happen once, but the row-to-Arrow rebuild is paid N times, and it is a rows x cols scalar copy each time.

`AuronColumnarRowData` exposes only per-field getters, so there is no zero-copy re-export of the underlying batch today, and the exporter has no already-Arrow-backed fast path.

Idea: hand Arrow batches across the operator edge instead of a per-row RowData stream, so a shared native source can feed multiple native Calc operators without rebuilding the columnar buffers per consumer. This is a batch-oriented operator exchange and is larger than the multi-consumer-safety scope of #2329, hence a separate issue.

Magnitude is unmeasured. A profiler run on a `source -> single Calc` job (parallelism 1) would quantify the CPU share spent in `FlinkArrowWriter.write` / `ArrowFieldWriter` versus Kafka read and native compute, to confirm the win is worth the change.

Contributor guide

Open the contributing guide

Research direction

Start with a profiler run on a parallelism-1 source-to-single-Calc job, focusing on FlinkArrowWriter.write and ArrowFieldWriter versus Kafka read and native compute. Then trace AuronColumnarRowData, FlinkAuronCalcOperator, FlinkArrowFFIExporter, and FlinkArrowWriter to understand the current row-to-Arrow path. Done means shared native sources can feed multiple native Calcs without rebuilding columnar buffers per consumer, with measurements showing the resulting trade-off.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.