apache / apache/texera

NaN keys crash Python range shuffle

Open
#8,257 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

The JVM range partitioner converts `Double.NaN` to long zero before routing, while the Python partitioner performs arithmetic on NaN and raises `ValueError`. A workflow containing a NaN range key therefore crashes only in the Python engine.

Before: NaN range key raises ValueError in Python

After: NaN is treated as zero and routed like the JVM

Python should treat a NaN range key as zero, matching JVM double-to-long conversion and routing it to receiver B for this range.

Reproduction evidence:

Range-shuffle a NaN key across receivers A and B with minimum -10 and maximum 9.

Observed result:

```text
ValueError: cannot convert float NaN to integer
scala_nan_to_long_index=1
```

Version and commit evidence:

1.3.0-incubating-SNAPSHOT (main)

**Commit Hash (Optional)**

`50321e403c82df299a13deb50a7f9849dd93bdba`

### Proposed Solution or Design

After: NaN is treated as zero and routed like the JVM

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.