apache / apache/arrow

[C++][JNI] DisposableScannerAdaptor does not handle arrays with offsets

Open
#30,767 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

The DisposableScannerAdaptor is a JNI bridge from Java to the C++ datasets API. When it scans record batches it collects all of the buffers from all of the arrays and returns a list of buffer handles to Java which puts these into an ArrowRecordBatch on the Java end.

Unfortunately, if the array has offsets then the bridge does not return the offset buffer but it returns the entire buffer. The Java record batch is then incorrect. The length is wrong (and so it doesn't fully free the memory) and the values are incorrect.

I'm not familiar enough with the Java implementation to suggest a good fix. Figuring out the buffer offsets from array offsets is a bit tricky since the logic depends on the data type. Also, I'm pretty sure the Java side now has to take ownership of the entire buffer which could be tricky because multiple batches could share ownership of the buffer.

As a temporary fix for ARROW-13554 I am going to copy the array if it has an offset. This means the transfer is not zero-copy so I'm creating this issue to solve this properly.

**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-15275) / @westonpace
#### Related issues:
- [[Java] Dataset JNI bridge should use the C data interface](https://github.com/apache/arrow/issues/31199) (is superceded by)

**Note**: *This issue was originally created as [ARROW-15275](https://issues.apache.org/jira/browse/ARROW-15275). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by tracing DisposableScannerAdaptor and the Java ArrowRecordBatch buffer handling, then review the related superseding C data interface issue. Done means offset arrays produce correct Java record batches through zero-copy transfer with safe buffer ownership.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
api, backend-api-design
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.