IndexOutOfBoundsException in UDAF
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 21
Description
**Describe the bug**
When executing Spark UDAFs, an `IndexOutOfBoundsException` occurs on the Java side during the importRows call. This happens when the native engine passes a sliced Arrow array to Java via FFI. The Rust implementation was passing the sliced array directly, which retains the original buffer and non-zero offsets. The Java consumer, however, expects the Arrow array to be self-contained with 0-based offsets, leading to an out-of-bounds access when it tries to read from the buffer using the raw offsets.
**Additional context**
```
Exception in thread "auron native task 331.0 in stage 6.0 (TID 1693)" java.lang.IndexOutOfBoundsException: index: 1068032, length: 120 (expected: range(0, 192600))
at auron.org.apache.arrow.memory.ArrowBuf.checkIndex(ArrowBuf.java:702)
at auron.org.apache.arrow.memory.ArrowBuf.getBytes(ArrowBuf.java:729)
at auron.org.apache.arrow.vector.VarBinaryVector.get(VarBinaryVector.java:112)
at org.apache.spark.sql.auron.TypedImperativeEvaluator.$anonfun$importRows$6(SparkUDAFWrapperContext.scala:451)
```
Contributor guide
Assessment
This issue has not been assessed yet.