apache / apache/texera

Raw binary values with a pickle prefix fail Arrow decoding

Open
#8,268 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

`ArrowTableTupleProvider` treats every binary value beginning with the six bytes `pickle` as Texera's pickled-object envelope. The actual envelope uses the ten-byte header `pickle` followed by four spaces. Legitimate raw bytes such as `b"picklehello"` are therefore passed to `pickle.loads` and fail instead of round-tripping unchanged.

Before: a raw binary value beginning with `pickle` can raise `UnpicklingError` while being read from Arrow.

Expected: only the complete Texera pickle header triggers unpickling; other binary values remain bytes.

Reproduction evidence:

1. Build an Arrow binary column containing `b"picklehello"`.
2. Read it through `ArrowTableTupleProvider`.
3. Access the field and observe `_pickle.UnpicklingError`.

Version and commit evidence:

1.3.0-incubating-SNAPSHOT on main

**Commit Hash (Optional)**
50321e403c82df299a13deb50a7f9849dd93bdba

**Relevant log output**

_pickle.UnpicklingError: could not find MARK

### Proposed Solution or Design

Expected: only the complete Texera pickle header triggers unpickling; other binary values remain bytes.

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.