datafusion-contrib / datafusion-contrib/StreamFusion
Native retracting OFFSET: preserve upstream stateful mini-batch changelog order
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 120
- Forks
- 8
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 76
Description
Remaining problem
Hidden-rank retracting OFFSET still falls back when an upstream stateful mini-batch operator can reorder or collapse its input changelog. Released Flink retains emitted row kinds inside the rank buffer, so those input-order differences can affect later full-row retraction matching and the materialized result.
The retained-row-kind implementation shipped in PR #132, merged as 40290ec9. It preserves sort-key counts independently of retained payloads, emits the per-record cascade, honors both update-before settings, and restores retained kinds/counts across memory and RocksDB. The admitted path passed exact ordered changelog comparisons, the full Java/Rust suites, unchanged upstream Rank/Limit/SortLimit tests, and all 24 required PR checks.
Current coverage and release measurements live in LIMIT and Top-N. Unique-key update-fast OFFSET and projected-rank paths remain covered separately.
Remaining scope
- Reproduce the upstream stateful mini-batch changelog order needed by hidden-rank retracting OFFSET, including grouped pipelines whose bundles collapse intermediate changes.
- Preserve full-row retraction matching, retained row kinds and independent sort-key counts through that composition.
- Keep the precise
retracting OFFSET requires unchanged upstream mini-batch changelog ordergate until the full pipeline is verified.
Acceptance
- Compare exact raw changelogs and final materializations against released Flink with stateful upstream mini-batches, duplicate payloads, tied keys, NULL ordering and group deletion.
- Cover both update-before settings and retractions after prior positional emission.
- Verify checkpoint continuation and both state backends for the admitted composition.
- Assert native execution of the full grouped/OFFSET pipeline and update the LIMIT/Top-N coverage pages when the remaining gate is removed.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the LIMIT and Top-N coverage pages and the existing native retracting OFFSET gate. Reproduce the grouped stateful mini-batch composition, compare exact raw changelogs and final materializations with released Flink, then verify checkpoint continuation, both state backends, update-before settings, native execution, and the coverage-page updates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, rust
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100