High binary bytes can route to different hash partitions
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Feature Summary
Python exposes byte values as unsigned integers, but Java byte arrays hash signed byte values. Binary tuple fields containing bytes above `0x7f` therefore produce different hashes in the Python and JVM engines, which can route the same hash-shuffle key to different workers.
Before: the `0xff` key hashes to 317 in Python and 61 with Java semantics
After: both engines hash the key to 61
Python binary tuple hashing should convert bytes to signed Java values before applying the array hash so shuffle routing matches the JVM.
Reproduction evidence:
Hash a tuple with one binary field containing byte `0xff`.
Observed values:
```text
python_tuple_hash=317
java_compatible_tuple_hash=61
```
Version and commit evidence:
1.3.0-incubating-SNAPSHOT (main)
**Commit Hash (Optional)**
`50321e403c82df299a13deb50a7f9849dd93bdba`
### Proposed Solution or Design
After: both engines hash the key to 61
### Affected Area
Workflow Engine (Amber)
Contributor guide
Assessment
This issue has not been assessed yet.