apache / apache/texera

Broadcast flush drops pending tuples for later receivers

Open
#8,189 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

The broadcast partitioner stores one shared pending batch, but a targeted flush resets that batch after the first receiver. OutputManager flushes control boundaries one receiver at a time, so only the first receiver receives pending tuples. Later receivers receive the control message without the tuples. Flushing an unknown receiver also clears pending data.

Before: flush A sends tuple and marker, then flush B sends only the marker.

After: each configured receiver gets the tuple once before the shared batch is cleared. Repeated flushes do not duplicate it, and unknown receivers leave it untouched.

Reproduction evidence:

1. Create a BroadcastPartitioner with receivers A and B.
2. Add one tuple below the configured batch size.
3. Flush a control message to A, then flush the same boundary to B.
4. Observe that A receives the tuple and marker while B receives only the marker.
5. Add another pending tuple and flush an unknown receiver. Observe that the tuple is discarded.

Version and commit evidence:

1.3.0-incubating-SNAPSHOT (main)

**Commit Hash (Optional)**
70c21145887920528d7d5540e3fb790b43e8b759

## Relevant log output

receiver_A=[1, EmbeddedControlMessage]
receiver_B=[EmbeddedControlMessage]
batch_after_unknown=0

### Proposed Solution or Design

After: each configured receiver gets the tuple once before the shared batch is cleared. Repeated flushes do not duplicate it, and unknown receivers leave it untouched.

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.