[C++][JNI] DisposableScannerAdaptor does not handle arrays with offsets
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
The DisposableScannerAdaptor is a JNI bridge from Java to the C++ datasets API. When it scans record batches it collects all of the buffers from all of the arrays and returns a list of buffer handles to Java which puts these into an ArrowRecordBatch on the Java end.
Unfortunately, if the array has offsets then the bridge does not return the offset buffer but it returns the entire buffer. The Java record batch is then incorrect. The length is wrong (and so it doesn't fully free the memory) and the values are incorrect.
I'm not familiar enough with the Java implementation to suggest a good fix. Figuring out the buffer offsets from array offsets is a bit tricky since the logic depends on the data type. Also, I'm pretty sure the Java side now has to take ownership of the entire buffer which could be tricky because multiple batches could share ownership of the buffer.
As a temporary fix for ARROW-13554 I am going to copy the array if it has an offset. This means the transfer is not zero-copy so I'm creating this issue to solve this properly.
**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-15275) / @westonpace
#### Related issues:
- [[Java] Dataset JNI bridge should use the C data interface](https://github.com/apache/arrow/issues/31199) (is superceded by)
**Note**: *This issue was originally created as [ARROW-15275](https://issues.apache.org/jira/browse/ARROW-15275). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
No files or tests are named. Start by tracing DisposableScannerAdaptor and the Java ArrowRecordBatch buffer handling, then review the related superseding C data interface issue. Done means offset arrays produce correct Java record batches through zero-copy transfer with safe buffer ownership.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- api, backend-api-design
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100