rust: ADBC driver panics when consuming Arrow batches without properly aligned buffers
- Dominant language
- C#
- Stars
- 627
- Forks
- 217
- Avg merge
- 17h
- Merged PRs (30d)
- 57
Description
### What happened?
Running SQL queries through the Snowflake ADBC driver, from Rust, can cause a crash due to buffer alignment issues.
```
thread 'main' panicked at ~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-buffer-53.4.0/src/buffer/scalar.rs:143:17:
Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type. Before importing buffer through FFI, please make sure the allocation is aligned.
```
This is known problem for Rust [1] when it's consuming buffers from systems that don't care as much about alignment as `rustc` does. It's been fixed in the FFI integration with Python [2].
The ADBC Rust driver wrapper should do the same. How can it trigger the `ArrayData::align_buffers()` call that makes the Python FFI integration work without alignment issues?
[1] https://github.com/apache/arrow/issues/43552
[2] https://github.com/apache/arrow-rs/pull/6472
### Stack Trace
```
thread 'tokio-runtime-worker' panicked at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-buffer/src/buffer/scalar.rs:138:17:
Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type. Before importing buffer through FFI, please make sure the allocation is aligned.
stack backtrace:
0: rust_begin_unwind
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:662:5
1: core::panicking::panic_fmt
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:74:14
2: as core::convert::From>::from
3: >::into
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/convert/mod.rs:759:9
4: arrow_buffer::buffer::scalar::ScalarBuffer::new
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-buffer/src/buffer/scalar.rs:72:9
5: as core::convert::From>::from
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/array/primitive_array.rs:1538:22
6: arrow_array::array::make_array
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/array/mod.rs:793:48
7: >::from::{{closure}}
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/array/struct_array.rs:302:23
8: core::iter::adapters::map::map_fold::{{closure}}
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/adapters/map.rs:88:28
9: as core::iter::traits::iterator::Iterator>::fold
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/slice/iter/macros.rs:232:27
10: as core::iter::traits::iterator::Iterator>::fold
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/adapters/map.rs:128:9
11: core::iter::traits::iterator::Iterator::for_each
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/traits/iterator.rs:813:9
12: alloc::vec::Vec::extend_trusted
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/mod.rs:3121:17
13: as alloc::vec::spec_extend::SpecExtend>::spec_extend
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_extend.rs:26:9
14: as alloc::vec::spec_from_iter_nested::SpecFromIterNested>::from_iter
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_from_iter_nested.rs:60:9
15: as alloc::vec::spec_from_iter::SpecFromIter>::from_iter
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_from_iter.rs:33:9
16: as core::iter::traits::collect::FromIterator>::from_iter
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/mod.rs:2985:9
17: core::iter::traits::iterator::Iterator::collect
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/traits/iterator.rs:2000:9
18: >::from
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/array/struct_array.rs:299:22
19: ::next::{{closure}}
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/ffi_stream.rs:367:54
20: core::result::Result::map
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/result.rs:771:25
21: ::next
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/ffi_stream.rs:367:18
22: as core::iter::traits::iterator::Iterator>::next
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/adapters/map.rs:107:9
23: as alloc::vec::spec_from_iter_nested::SpecFromIterNested>::from_iter
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_from_iter_nested.rs:24:32
24: as alloc::vec::spec_from_iter::SpecFromIter>::from_iter
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_from_iter.rs:33:9
25: as core::iter::traits::collect::FromIterator>::from_iter
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/mod.rs:2985:9
26: core::iter::traits::iterator::Iterator::collect
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/traits/iterator.rs:2000:9
```
### How can we reproduce the bug?
_No response_
### Environment/Setup
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.