Unicode strings can route to different hash partitions
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Feature Summary
Python tuple hashing iterates Unicode code points, while Java String hashing iterates UTF-16 code units. Characters outside the basic multilingual plane therefore produce different tuple hashes in the Python and JVM engines, which can route the same hash-shuffle key to different workers.
Before: the emoji key hashes to 128543 in Python and 1772930 with Java semantics
After: both engines hash the key to 1772930
Python string tuple hashing should iterate UTF-16 code units so hash shuffle routing matches the JVM for every valid Unicode string.
Reproduction evidence:
Hash a tuple with one string field containing `😀`.
Observed values:
```text
python_tuple_hash=128543
java_compatible_tuple_hash=1772930
```
Version and commit evidence:
1.3.0-incubating-SNAPSHOT (main)
**Commit Hash (Optional)**
`50321e403c82df299a13deb50a7f9849dd93bdba`
### Proposed Solution or Design
After: both engines hash the key to 1772930
### Affected Area
Workflow Engine (Amber)
Contributor guide
Assessment
This issue has not been assessed yet.