Raw binary values with a pickle prefix fail Arrow decoding
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Feature Summary
`ArrowTableTupleProvider` treats every binary value beginning with the six bytes `pickle` as Texera's pickled-object envelope. The actual envelope uses the ten-byte header `pickle` followed by four spaces. Legitimate raw bytes such as `b"picklehello"` are therefore passed to `pickle.loads` and fail instead of round-tripping unchanged.
Before: a raw binary value beginning with `pickle` can raise `UnpicklingError` while being read from Arrow.
Expected: only the complete Texera pickle header triggers unpickling; other binary values remain bytes.
Reproduction evidence:
1. Build an Arrow binary column containing `b"picklehello"`.
2. Read it through `ArrowTableTupleProvider`.
3. Access the field and observe `_pickle.UnpicklingError`.
Version and commit evidence:
1.3.0-incubating-SNAPSHOT on main
**Commit Hash (Optional)**
50321e403c82df299a13deb50a7f9849dd93bdba
**Relevant log output**
_pickle.UnpicklingError: could not find MARK
### Proposed Solution or Design
Expected: only the complete Texera pickle header triggers unpickling; other binary values remain bytes.
### Affected Area
Workflow Engine (Amber)
Contributor guide
Assessment
This issue has not been assessed yet.