NaN payloads can route to different hash partitions
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Feature Summary
Java `Double` hashing canonicalizes all NaN bit patterns, while Python tuple hashing preserves the raw NaN payload. Noncanonical NaNs therefore produce different tuple hashes in the Python and JVM engines and can route the same hash-shuffle key to different workers.
Before: signaling NaN payload hashes to 2146435104 in Python and 2146959391 with Java semantics
After: every NaN hashes to the Java canonical value 2146959391
Expected behavior:
Python should canonicalize NaN bits before applying the Java double hash so shuffle routing matches the JVM.
Reproduction evidence:
On `50321e403c82df299a13deb50a7f9849dd93bdba`, create a double from bits `7ff0000000000001` and hash a tuple containing it.
Observed values:
```text
payload_bits=7ff0000000000001
python_tuple_hash=2146435104
java_compatible_tuple_hash=2146959391
```
Version and commit evidence:
1.3.0-incubating-SNAPSHOT (main)
**Commit Hash (Optional)**
`50321e403c82df299a13deb50a7f9849dd93bdba`
### Proposed Solution or Design
After: every NaN hashes to the Java canonical value 2146959391
Expected behavior:
### Affected Area
Workflow Engine (Amber)
Contributor guide
Assessment
This issue has not been assessed yet.