apache / apache/beam

Portable representation of "packed bitset indicating null fields" in beam Row format is not compatible with jvm representations

Open
#21,232 0 comments 0 reactions 0 assignees View on GitHub
bug cross-language P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The JVM RowCoder strips trailing 0s from the null-value bitmap, while both python and go expect all bits to be present in the encoded bitmap.  This causes index-out-of-range errors when trying to decode a row encoded on the JVM in other languages in some circumstances.

For example, given a Row with 10 nullable fields, if the first 8 are null and the last two are set, the row will fail to decode in python, because the nullable bitmap will only have 1 byte, but the python coder expects 2.

As discussed in the thread, the best solution here is probably to change the python (and go) coders to accept truncated nullable bitmaps.

 

More discussion here:

[https://lists.apache.org/thread.html/r2f148e29902bda8bb0ff7106fffb8a5494295450827ad7fd17289383%40%3Cdev.beam.apache.org%3E](https://lists.apache.org/thread.html/r2f148e29902bda8bb0ff7106fffb8a5494295450827ad7fd17289383%40%3Cdev.beam.apache.org%3E)

Imported from Jira [BEAM-13081](https://issues.apache.org/jira/browse/BEAM-13081). Original Jira may contain additional context.
Reported by: SteveNiemitz.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.