datafusion-contrib / datafusion-contrib/StreamFusion
Native Top-N: support nullable and changing variable rank bounds
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 120
- Forks
- 8
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 76
Description
Remaining implementation
- Nullable variable bounds, across all rank strategies.
- Changing update-fast bounds with positive idle-state TTL, including expiry while ranked rows remain live.
- Preserving the first proposal through upstream stateful mini-batches that can reorder a general retracting changelog.
Flink 2.2.1 holds the first bound in per-key rankEnd state. Later different proposals increment topn.invalidTopSize without resizing the window. Reads do not refresh the bound's TTL, so it can expire while ranked rows remain live. Primitive SMALLINT/INT/BIGINT access makes nullable bounds a separate compatibility concern.
Shipped coverage
Non-null partition-invariant bounds run natively for append-only, general retracting and update-fast input (8750beaf, #159 and #170). PR #179 ships proofs for complete computed partition keys, including arithmetic, MOD, integral casts, built-in COALESCE and composites. These proofs avoid additional bound state because recreation must produce the same value. See Top-N coverage.
Shipped changing-bound coverage
PR #180 shipped append-only first-bound state in c744ce0c; PR #181 shipped general retracting first-bound state in ec923f0d. Both merged after all 25 CI checks passed, including every required unchanged upstream suite. The first bound, its independent TTL and mismatch counter semantics survive checkpoints, canonical memory/RocksDB transitions and rescaling; full-row payloads retain the proposed bound. Deleting the final ranked row does not reset the bound. Raw changelog sources remain native under mini-batching, while upstream stateful bundles that can reorder proposals still fall back.
PR #201 ships non-null independently changing update-fast bounds with idle-state TTL disabled. It preserves the original proposal, mismatch counts and payloads across checkpoints, canonical memory/RocksDB transitions and 1→2→1 rescaling, including bound-only groups and nonpositive bounds. Positive TTL retains an explicit fallback because Flink refreshes dirty cached rows independently of the bound. All required CI checks passed before merging; the unchanged upstream Rank state selection also passed with native/fallback evidence. The recorded release+mimalloc workload measured 4.814× Flink with both transposes.
See Top-N coverage for the supported contract, regression coverage and release benchmarks (3.005x append-only and 1.912x retracting in the recorded row-fed workloads).
Remaining acceptance
- Match the update-fast cache's dirty-row TTL refresh and independent bound-expiry semantics before admitting changing bounds with positive retention.
- Preserve first-proposal ordering through upstream stateful mini-batches before admitting those compositions.
- Establish nullable-bound behavior across generated and materialized row representations before widening admission.
- For each newly admitted strategy, verify nonpositive/large bounds, ties, NULL payloads, expiry with live rows, mini-batch composition, checkpoints, rescaling and memory/RocksDB transitions.
- Run relevant unchanged upstream classes, assert actual native execution, and update the coverage page in the shipping commit.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Top-N coverage page and the shipped changing-bound work in PRs #180, #181, and #201, then inspect the upstream Rank state selection mentioned in the issue. Done requires native execution for the newly admitted cases, unchanged upstream suites and CI passing, coverage updates, and validation of TTL, nullable bounds, ordering, checkpoints, rescaling, and storage transitions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100