apache / apache/texera

Hash shuffle crashes on large binary fields

Open
#8,243 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Python tuple hashing has no mapping for the supported `LARGE_BINARY` attribute type. A hash shuffle that selects a large binary field therefore crashes before routing the tuple.

Before: hash a tuple with a large binary field and receive a KeyError

After: hash the stored URI with the same nested Java hash used by `LargeBinary` and `Tuple`

Expected behavior:

Large binary fields should be hashable for shuffle partitioning and produce the same stable value as the Java tuple implementation.

Reproduction evidence:

On `50321e403c82df299a13deb50a7f9849dd93bdba`, create a tuple with schema `{blob: LARGE_BINARY}` and value `s3://bucket/key`, then call `hash` on it.

Observed result:

```text
KeyError:
```

Version and commit evidence:

1.3.0-incubating-SNAPSHOT (main)

**Commit Hash (Optional)**

`50321e403c82df299a13deb50a7f9849dd93bdba`

### Proposed Solution or Design

After: hash the stored URI with the same nested Java hash used by `LargeBinary` and `Tuple`

Expected behavior:

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.