apache / apache/texera

Python Unicode hashes disagree with Scala for supplementary characters

Open
#8,181 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Java string hashing iterates UTF-16 code units, while Python tuple hashing iterates Unicode code points. These representations differ for supplementary characters, so equal string keys can choose different hash-shuffle workers in Python and Scala.

Expected behavior:

Python string tuple hashes use Java UTF-16 code-unit semantics.

Reproduction evidence:

Use two receivers and the string key represented by `U+1F600`.

Before: Python hash 128543 selects worker 1, Scala hash 1772930 selects worker 0
After: both hashes are 1772930 and both select worker 0

ASCII, BMP Unicode, and supplementary Unicode are covered.

Version and commit evidence:

Current `main` at 70c21145887920528d7d5540e3fb790b43e8b759.

**Commit Hash (Optional)**

`70c21145887920528d7d5540e3fb790b43e8b759`

### Proposed Solution or Design

Expected behavior:

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.