apache / apache/texera

Hash and range flushes resend pending tuples

Open
#8,187 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Python hash and range partitioners emit pending batches during `flush` and `flush_state` but do not clear those batches afterward. Later control markers can resend the same tuples, unlike the round-robin, broadcast, and one-to-one partitioners.

Expected behavior:

A batch emitted across a control or state boundary is cleared exactly once.

Reproduction evidence:

Add one tuple below the batch-size threshold, flush its receiver twice, and inspect both outputs.

Before: first flush sends the tuple and marker, second flush sends the same tuple and marker again
After: first flush sends the tuple and marker, second flush sends only the marker

The same stale-batch behavior occurs after `flush_state` in both partitioners.

Version and commit evidence:

Current `main` at 70c21145887920528d7d5540e3fb790b43e8b759.

**Commit Hash (Optional)**

`70c21145887920528d7d5540e3fb790b43e8b759`

### Proposed Solution or Design

Expected behavior:

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.