FixedSizeListArray::value_offset wraps past i32::MAX, and the avro writer panics on it
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
### Describe the bug
`FixedSizeListArray::value_offset` works the offset out in usize and then narrows it with `as i32`. Once a row starts past i32::MAX child values it returns a negative number with no error, while `value(i)` on the same row is still right because it never leaves usize.
arrow-avro's FixedSizeList encoder reads it as `self.list.value_offset(idx) as usize` (writer/encoder.rs:2129), so the negative offset turns into an index near u64::MAX and writing the batch panics.
### To Reproduce
arrow-array, arrow-buffer, arrow-schema and arrow-avro 59.3.0, arrow-avro with default-features = false.
```rust
use std::sync::Arc;
use arrow_array::{ArrayRef, BooleanArray, FixedSizeListArray, RecordBatch};
use arrow_avro::writer::AvroWriter;
use arrow_buffer::{BooleanBuffer, NullBuffer};
use arrow_schema::{DataType, Field, Schema};
fn main() {
let size: i32 = 1 << 30;
let values = BooleanArray::new(BooleanBuffer::new_unset(size as usize * 3), None);
let item = Arc::new(Field::new_list_field(DataType::Boolean, false));
let nulls = NullBuffer::from(vec![false, false, true]);
let list = FixedSizeListArray::try_new(item.clone(), size, Arc::new(values), Some(nulls)).unwrap();
println!("row 2: value_offset = {}, actual offset = {}", list.value_offset(2), 2 * size as usize);
let schema = Schema::new(vec![Field::new("v", DataType::FixedSizeList(item, size), true)]);
let batch = RecordBatch::try_new(Arc::new(schema.clone()), vec![Arc::new(list) as ArrayRef]).unwrap();
let mut writer = AvroWriter::new(std::io::sink(), schema).unwrap();
writer.write(&batch).unwrap();
}
```
```
row 2: value_offset = -2147483648, actual offset = 2147483648
thread 'main' panicked at arrow-array-59.3.0/src/array/boolean_array.rs:245:9:
Trying to access an element at index 18446744071562067968 from a BooleanArray of length 3221225472
```
Rows 0 and 1 are null so only row 2 gets encoded. The child is 384 MiB of booleans, so it runs on a laptop.
### Expected behavior
The batch writes. value_offset either returns the real offset or isn't there to call.
### Additional context
FixedSizeBinaryArray::value_offset had the same narrowing and was deprecated in 59.0.0 in favour of `i * value_size()`. The FixedSizeList one could go the same way, with the avro encoder using `idx * value_length() as usize` instead. The only other non-test caller is take_value_indices_from_fixed_size_list, which casts the result to u32, so it comes out right up to u32::MAX, where its UInt32 indices run out anyway.
Happy to send the PR if that direction works.
Contributor guide
Assessment
This issue has not been assessed yet.