apache / apache/arrow-rs

FixedSizeListArray::value_offset wraps past i32::MAX, and the avro writer panics on it

Open
#11,059 1 comment 0 reactions 1 assignee Claimed by @imtherealnaska View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

### Describe the bug

`FixedSizeListArray::value_offset` works the offset out in usize and then narrows it with `as i32`. Once a row starts past i32::MAX child values it returns a negative number with no error, while `value(i)` on the same row is still right because it never leaves usize.

arrow-avro's FixedSizeList encoder reads it as `self.list.value_offset(idx) as usize` (writer/encoder.rs:2129), so the negative offset turns into an index near u64::MAX and writing the batch panics.

### To Reproduce

arrow-array, arrow-buffer, arrow-schema and arrow-avro 59.3.0, arrow-avro with default-features = false.

```rust
use std::sync::Arc;

use arrow_array::{ArrayRef, BooleanArray, FixedSizeListArray, RecordBatch};
use arrow_avro::writer::AvroWriter;
use arrow_buffer::{BooleanBuffer, NullBuffer};
use arrow_schema::{DataType, Field, Schema};

fn main() {
let size: i32 = 1 << 30;
let values = BooleanArray::new(BooleanBuffer::new_unset(size as usize * 3), None);
let item = Arc::new(Field::new_list_field(DataType::Boolean, false));
let nulls = NullBuffer::from(vec![false, false, true]);
let list = FixedSizeListArray::try_new(item.clone(), size, Arc::new(values), Some(nulls)).unwrap();
println!("row 2: value_offset = {}, actual offset = {}", list.value_offset(2), 2 * size as usize);

let schema = Schema::new(vec![Field::new("v", DataType::FixedSizeList(item, size), true)]);
let batch = RecordBatch::try_new(Arc::new(schema.clone()), vec![Arc::new(list) as ArrayRef]).unwrap();
let mut writer = AvroWriter::new(std::io::sink(), schema).unwrap();
writer.write(&batch).unwrap();
}
```

```
row 2: value_offset = -2147483648, actual offset = 2147483648
thread 'main' panicked at arrow-array-59.3.0/src/array/boolean_array.rs:245:9:
Trying to access an element at index 18446744071562067968 from a BooleanArray of length 3221225472
```

Rows 0 and 1 are null so only row 2 gets encoded. The child is 384 MiB of booleans, so it runs on a laptop.

### Expected behavior

The batch writes. value_offset either returns the real offset or isn't there to call.

### Additional context

FixedSizeBinaryArray::value_offset had the same narrowing and was deprecated in 59.0.0 in favour of `i * value_size()`. The FixedSizeList one could go the same way, with the avro encoder using `idx * value_length() as usize` instead. The only other non-test caller is take_value_indices_from_fixed_size_list, which casts the result to u32, so it comes out right up to u32::MAX, where its UInt32 indices run out anyway.

Happy to send the PR if that direction works.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.