apache / apache/texera

High binary bytes can route to different hash partitions

Open
#8,249 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Python exposes byte values as unsigned integers, but Java byte arrays hash signed byte values. Binary tuple fields containing bytes above `0x7f` therefore produce different hashes in the Python and JVM engines, which can route the same hash-shuffle key to different workers.

Before: the `0xff` key hashes to 317 in Python and 61 with Java semantics

After: both engines hash the key to 61

Python binary tuple hashing should convert bytes to signed Java values before applying the array hash so shuffle routing matches the JVM.

Reproduction evidence:

Hash a tuple with one binary field containing byte `0xff`.

Observed values:

```text
python_tuple_hash=317
java_compatible_tuple_hash=61
```

Version and commit evidence:

1.3.0-incubating-SNAPSHOT (main)

**Commit Hash (Optional)**

`50321e403c82df299a13deb50a7f9849dd93bdba`

### Proposed Solution or Design

After: both engines hash the key to 61

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.