datafusion-contrib / datafusion-contrib/StreamFusion

Native retracting OFFSET: preserve upstream stateful mini-batch changelog order

Open
#102 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
120
Forks
8
Avg merge
7h 29m
Merged PRs (30d)
76

Description

Remaining problem

Hidden-rank retracting OFFSET still falls back when an upstream stateful mini-batch operator can reorder or collapse its input changelog. Released Flink retains emitted row kinds inside the rank buffer, so those input-order differences can affect later full-row retraction matching and the materialized result.

The retained-row-kind implementation shipped in PR #132, merged as 40290ec9. It preserves sort-key counts independently of retained payloads, emits the per-record cascade, honors both update-before settings, and restores retained kinds/counts across memory and RocksDB. The admitted path passed exact ordered changelog comparisons, the full Java/Rust suites, unchanged upstream Rank/Limit/SortLimit tests, and all 24 required PR checks.

Current coverage and release measurements live in LIMIT and Top-N. Unique-key update-fast OFFSET and projected-rank paths remain covered separately.

Remaining scope

  • Reproduce the upstream stateful mini-batch changelog order needed by hidden-rank retracting OFFSET, including grouped pipelines whose bundles collapse intermediate changes.
  • Preserve full-row retraction matching, retained row kinds and independent sort-key counts through that composition.
  • Keep the precise retracting OFFSET requires unchanged upstream mini-batch changelog order gate until the full pipeline is verified.

Acceptance

  • Compare exact raw changelogs and final materializations against released Flink with stateful upstream mini-batches, duplicate payloads, tied keys, NULL ordering and group deletion.
  • Cover both update-before settings and retractions after prior positional emission.
  • Verify checkpoint continuation and both state backends for the admitted composition.
  • Assert native execution of the full grouped/OFFSET pipeline and update the LIMIT/Top-N coverage pages when the remaining gate is removed.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the LIMIT and Top-N coverage pages and the existing native retracting OFFSET gate. Reproduce the grouped stateful mini-batch composition, compare exact raw changelogs and final materializations with released Flink, then verify checkpoint continuation, both state backends, update-before settings, native execution, and the coverage-page updates.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, rust
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.