apache / apache/texera

Unicode strings can route to different hash partitions

Open
#8,247 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Python tuple hashing iterates Unicode code points, while Java String hashing iterates UTF-16 code units. Characters outside the basic multilingual plane therefore produce different tuple hashes in the Python and JVM engines, which can route the same hash-shuffle key to different workers.

Before: the emoji key hashes to 128543 in Python and 1772930 with Java semantics

After: both engines hash the key to 1772930

Python string tuple hashing should iterate UTF-16 code units so hash shuffle routing matches the JVM for every valid Unicode string.

Reproduction evidence:

Hash a tuple with one string field containing `😀`.

Observed values:

```text
python_tuple_hash=128543
java_compatible_tuple_hash=1772930
```

Version and commit evidence:

1.3.0-incubating-SNAPSHOT (main)

**Commit Hash (Optional)**

`50321e403c82df299a13deb50a7f9849dd93bdba`

### Proposed Solution or Design

After: both engines hash the key to 1772930

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.