apache / apache/texera

Python range shuffle crashes when channels repeat a receiver

Open
#8,169 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

The Python range partitioner deduplicates downstream receivers when it builds its batches, but calculates the range width from the original channel count. When multiple channels point to the same worker, the calculated receiver index can exceed the deduplicated receiver list.

Before: channels A, A, B over range 0 through 8 route value 8 to index 2 in a two-receiver list and raise `IndexError`.

After: the range width uses the two distinct receivers, so value 8 routes to receiver B at index 1.

The Scala range partitioner already calculates its range width from distinct receivers.

Reproduction evidence:

1. Construct a Python `RangeBasedShufflePartitioner` with channels to A, A, and B.
2. Set the range to 0 through 8.
3. Add a tuple whose range key is 8.
4. Observe the out-of-range receiver lookup.

Version and commit evidence:

1.3.0-incubating-SNAPSHOT on main

**Commit Hash (Optional)**
70c21145887920528d7d5540e3fb790b43e8b759

**Relevant log output**

IndexError: list index out of range

### Proposed Solution or Design

After: the range width uses the two distinct receivers, so value 8 routes to receiver B at index 1.

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.