apache / apache/arrow-rs

`safe: false` casting doesn't account for masked null values

Open
#10,711 2 comments 0 reactions 1 assignee Claimed by @dhruvxvaishnav View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Describe the bug

if casting with `safe: false` then errors are raised for values that can't be cast. however if the array has values masked by nulls, these values still try to get cast and if they are invalid values can cause the entire cast to fail even though they aren't visible.

### To Reproduce

binary -> utf8:

```rust
// arrow-cast/src/cast/mod.rs

#[test]
fn test_binary_not_safe() {
let mut buffer = MutableBuffer::new(4);
buffer.extend_from_slice(&[b'a', b'b', 0xFF, 0xFF]);
let array = BinaryArray::new(
OffsetBuffer::from_lengths([2, 2]),
buffer.into(),
// [0xFF, 0xFF] element is masked by null
Some(vec![true, false].into()),
);
// Currently failing
cast_with_options(
&array,
&DataType::Utf8,
&CastOptions {
safe: false,
format_options: Default::default(),
},
)
.unwrap();
}
```

fixedsizelist -> fixedsizelist:

```rust
// arrow-cast/src/cast/mod.rs

#[test]
fn test_fsl_not_safe() {
// 2nd element masked by null
let child = StringArray::from(vec!["1", "notnumber"]);
let array = FixedSizeListArray::new(
Field::new("item", DataType::Utf8, false).into(),
1,
Arc::new(child),
Some(vec![true, false].into()),
);
// Currently failing
cast_with_options(
&array,
&DataType::FixedSizeList(Field::new("item", DataType::Int32, false).into(), 1),
&CastOptions {
safe: false,
format_options: Default::default(),
},
)
.unwrap();
}
```

### Expected behavior

ideally the above tests would succeed, and this likely needs to be fixed for other cast paths (e.g. other list container types probably suffer from same bug)

### Additional context

paths to consider/fix:

- [ ] binary to string
- [ ] list type to list type
- [ ] check what other paths might be susceptible to this

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.