apache / apache/texera

NaN payloads can route to different hash partitions

Open
#8,251 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Java `Double` hashing canonicalizes all NaN bit patterns, while Python tuple hashing preserves the raw NaN payload. Noncanonical NaNs therefore produce different tuple hashes in the Python and JVM engines and can route the same hash-shuffle key to different workers.

Before: signaling NaN payload hashes to 2146435104 in Python and 2146959391 with Java semantics

After: every NaN hashes to the Java canonical value 2146959391

Expected behavior:

Python should canonicalize NaN bits before applying the Java double hash so shuffle routing matches the JVM.

Reproduction evidence:

On `50321e403c82df299a13deb50a7f9849dd93bdba`, create a double from bits `7ff0000000000001` and hash a tuple containing it.

Observed values:

```text
payload_bits=7ff0000000000001
python_tuple_hash=2146435104
java_compatible_tuple_hash=2146959391
```

Version and commit evidence:

1.3.0-incubating-SNAPSHOT (main)

**Commit Hash (Optional)**

`50321e403c82df299a13deb50a7f9849dd93bdba`

### Proposed Solution or Design

After: every NaN hashes to the Java canonical value 2146959391

Expected behavior:

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.