`safe: false` casting doesn't account for masked null values
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
### Describe the bug
if casting with `safe: false` then errors are raised for values that can't be cast. however if the array has values masked by nulls, these values still try to get cast and if they are invalid values can cause the entire cast to fail even though they aren't visible.
### To Reproduce
binary -> utf8:
```rust
// arrow-cast/src/cast/mod.rs
#[test]
fn test_binary_not_safe() {
let mut buffer = MutableBuffer::new(4);
buffer.extend_from_slice(&[b'a', b'b', 0xFF, 0xFF]);
let array = BinaryArray::new(
OffsetBuffer::from_lengths([2, 2]),
buffer.into(),
// [0xFF, 0xFF] element is masked by null
Some(vec![true, false].into()),
);
// Currently failing
cast_with_options(
&array,
&DataType::Utf8,
&CastOptions {
safe: false,
format_options: Default::default(),
},
)
.unwrap();
}
```
fixedsizelist -> fixedsizelist:
```rust
// arrow-cast/src/cast/mod.rs
#[test]
fn test_fsl_not_safe() {
// 2nd element masked by null
let child = StringArray::from(vec!["1", "notnumber"]);
let array = FixedSizeListArray::new(
Field::new("item", DataType::Utf8, false).into(),
1,
Arc::new(child),
Some(vec![true, false].into()),
);
// Currently failing
cast_with_options(
&array,
&DataType::FixedSizeList(Field::new("item", DataType::Int32, false).into(), 1),
&CastOptions {
safe: false,
format_options: Default::default(),
},
)
.unwrap();
}
```
### Expected behavior
ideally the above tests would succeed, and this likely needs to be fixed for other cast paths (e.g. other list container types probably suffer from same bug)
### Additional context
paths to consider/fix:
- [ ] binary to string
- [ ] list type to list type
- [ ] check what other paths might be susceptible to this
Contributor guide
Assessment
This issue has not been assessed yet.