apache / apache/texera

Python hash normalization can change shuffle receivers

Open
#8,253 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Python reserves hash value `-1` and rewrites it to `-2` when the `hash` builtin calls an object's hash method. Texera's tuple method can correctly calculate Java hash `-1`, but the Python hash shuffle passes it through the builtin and selects a different receiver from the JVM.

Before: Java tuple hash -1 becomes Python hash -2 and routes to receiver A

After: the partitioner uses the raw tuple hash and routes to receiver B like the JVM

Expected behavior:

Hash shuffle should use the tuple's raw Java-compatible hash before floor-mod routing so Python's reserved hash normalization cannot change the partition.

Reproduction evidence:

On `50321e403c82df299a13deb50a7f9849dd93bdba`, hash-shuffle an integer key of `-32` across receivers A and B.

Observed result:

```text
raw_java_hash=-1 builtin_hash=-2 receiver=A
```

Version and commit evidence:

1.3.0-incubating-SNAPSHOT (main)

**Commit Hash (Optional)**

`50321e403c82df299a13deb50a7f9849dd93bdba`

### Proposed Solution or Design

Use the tuple's raw Java-compatible hash before floor-mod routing so Python's reserved -1 normalization cannot change the receiver.

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.