datafusion-contrib / datafusion-contrib/StreamFusion
Native window aggregation: support retracting input from upstream Top-N
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 120
- Forks
- 8
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 76
Description
Remaining work
Native signed COUNT/SUM windows after updating Top-N and CDC sources shipped in PR #150. Zero-offset event-time TUMBLE/HOP/CUMULATE support one- and two-phase execution, unfiltered integer SUM, numeric COUNT(value), and COUNT(*). Grouping-only windows and integer AVG shipped in PR #154; FLOAT/DOUBLE AVG shipped in PR #155, FLOAT/DOUBLE SUM shipped in PR #157, DECIMAL SUM shipped in PR #158, and DECIMAL AVG shipped in PR #161. Signed updates, zero-count partials, floating cancellation and memory/RocksDB recovery have released-Flink parity and native execution coverage. Replacement/removal, all-NULL live groups, unmatched deletes, local/global partial buffers, late input and checkpoint recovery have released-Flink parity coverage. Runtime metrics require native Top-N and every admitted window stage to process and emit rows.
Extend verified retracting admission beyond that initial subset:
- DISTINCT and MIN/MAX.
- Filtered DISTINCT and filtered aggregate/window forms beyond aligned event-time COUNT/SUM/AVG and append-only MIN/MAX. The aligned COUNT/SUM/AVG subset shipped in PR #174 (a8170594), with every required CI and upstream suite passing; both append-only and retracting input, one/two-phase execution, memory/RocksDB recovery and native runtime counters were verified.
- Processing-time, attached, session and legacy window forms.
- Optimizer rewrites that combine overlapping COUNT, SUM and AVG over the same value.
These shapes keep an explicit planner fallback. Preserve signed accumulator state, group liveness, late-window admission, cleanup, and local/global buffer layouts for each extension. The coverage source of truth is window aggregation.
Acceptance for each extension
- Compare controlled +I/-U/+U/-D inputs, duplicates, last-value removal and NULL behavior with released Flink.
- Verify final results and emitted changelog, late input and watermark firing.
- Exercise memory and RocksDB checkpoint/restore and any admitted local/global phase.
- Require actual native execution and retain a precise reason for unsupported forms.
- Update coverage documentation and measure the end-to-end release build with both transposes.
The initial COUNT/SUM pipeline is slower than Flink in the documented benchmarks; it provides a tested native changelog and recovery foundation, not a measured speedup.
PR #177 adds filtered numeric MIN/MAX for append-only aligned event-time TUMBLE/HOP/CUMULATE, in one- and two-phase execution. Local validation passed 231 Java tests and 36 unchanged upstream window executions, including native execution checks and memory/RocksDB restore. The release mixed-query benchmark was effectively tied in single-phase and 1.18x faster in local/global execution, including both transposes. It shipped as 81b5f00c after all 25 required CI checks passed, including every unchanged upstream suite. Retracting MIN/MAX remains open.
PR #178 implements filtered COUNT(DISTINCT) for append-only aligned event-time TUMBLE/HOP/CUMULATE across the existing exact numeric, string and temporal types. Full Java validation passed (3,329 cases, 33 skips, plus 24 ORC-specific checks), and all 88 selected unchanged upstream window executions passed with native/fallback contracts enforced. Tests cover independent filters, nullable groups, late watermarks and memory/RocksDB recovery in both aggregation phases. The mixed-query release benchmark measured 1.054x single-phase and 1.176x local/global versus Flink, including both transposes. All 25 CI checks passed, including the complete upstream runtime, formats, parquet, ORC, Kafka, Paimon, Delta and state suites. Merged as 13edaa7c; retracting DISTINCT remains open.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked window aggregation coverage documentation and the existing native COUNT/SUM/AVG, MIN/MAX, and DISTINCT validation described in this issue. Select one remaining extension, compare its controlled changelog behavior with released Flink, and verify late input, watermark firing, checkpoint recovery, phase execution, and native/fallback contracts. Done means the extension is covered, documented, and retains a precise fallback reason.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100