Python timestamp hashes disagree with Scala at subsecond precision
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Feature Summary
Python tuple hashing converts timestamp keys to whole epoch seconds, while Scala delegates to Java `Timestamp.hashCode`, which uses epoch milliseconds. Equal timestamp keys can therefore choose different workers in a hash shuffle when Scala and Python producers feed the same downstream operator.
Expected behavior:
Python tuple hashes match Java tuple hashes at timestamp millisecond precision.
Reproduction evidence:
Use three receivers and the timestamp `1970-01-01T00:00:00.500Z`.
Before: Python hash 31 selects worker 1, Scala hash 531 selects worker 0
After: both hashes are 531 and both select worker 0
Negative, zero, and positive millisecond boundaries are covered.
Version and commit evidence:
Current `main` at 70c21145887920528d7d5540e3fb790b43e8b759.
**Commit Hash (Optional)**
`70c21145887920528d7d5540e3fb790b43e8b759`
### Proposed Solution or Design
Expected behavior:
### Affected Area
Workflow Engine (Amber)
Contributor guide
Assessment
This issue has not been assessed yet.