apache / apache/texera

Python timestamp hashes disagree with Scala at subsecond precision

Open
#8,173 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Python tuple hashing converts timestamp keys to whole epoch seconds, while Scala delegates to Java `Timestamp.hashCode`, which uses epoch milliseconds. Equal timestamp keys can therefore choose different workers in a hash shuffle when Scala and Python producers feed the same downstream operator.

Expected behavior:

Python tuple hashes match Java tuple hashes at timestamp millisecond precision.

Reproduction evidence:

Use three receivers and the timestamp `1970-01-01T00:00:00.500Z`.

Before: Python hash 31 selects worker 1, Scala hash 531 selects worker 0
After: both hashes are 531 and both select worker 0

Negative, zero, and positive millisecond boundaries are covered.

Version and commit evidence:

Current `main` at 70c21145887920528d7d5540e3fb790b43e8b759.

**Commit Hash (Optional)**

`70c21145887920528d7d5540e3fb790b43e8b759`

### Proposed Solution or Design

Expected behavior:

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.