apache / apache/arrow-rs

Filtering items in list inside ListArray

Open
#6,846 15 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Which part is this question about**
library api

**Describe your question**
I have a `ListArray` and I want to filter each item in the list

```text
┌─────────────┐ -> ┌─────────────┐
│ [A,B,C] │ -> │ [A,C] │
├─────────────┤ -> ├─────────────┤
│ [] │ -> │ [] │
├─────────────┤ -> ├─────────────┤
│ NULL │ -> │ NULL │
├─────────────┤ -> ├─────────────┤
│ [D] │ -> │ [] │
├─────────────┤ -> ├─────────────┤
│ [NULL, F] │ -> │ [NULL, F] │
└─────────────┘ -> └─────────────┘
```

But I couldn't find a way to do it

I tried this:
```rust

#[cfg(test)]
mod tests {
use arrow::array::{Array, BooleanArray, ListArray};
use arrow::datatypes::Int32Type;
use arrow::error::ArrowError;
use std::ops::Deref;

#[test]
fn list_array() -> Result<(), ArrowError> {
// Create list
let list_array = create_list_for_test();

// go over each list in the array and filter items in it
let new_list: ListArray = list_array
.iter()
.map(|x| {
match x {
None => Ok(None),
Some(x) => {
let some_predicate = BooleanArray::from((0..x.len()).map(|i| i % 2 == 0).collect::>());
let val = arrow::compute::filter(x.deref(), &some_predicate)?;

Ok(Some(val))
}
}
})
.collect::>()?;

println!("{:?}", new_list);

Ok(())
}

// This can be list of list of int32 or any other inner list type
fn create_list_for_test() -> ListArray {
let data = vec![
Some(vec![Some(0), Some(1), Some(2)]),
None,
Some(vec![Some(3), None, Some(5)]),
Some(vec![Some(6), Some(7)]),
];
let list_array = ListArray::from_iter_primitive::(data);

list_array
}
}
```

But got:
```
error[E0277]: a value of type `GenericListArray` cannot be built from an iterator over elements of type `Option>`
--> arrow-pg/src/lib.rs:31:24
|
31 | .collect::>()?;
| ------- ^^^^^^^^^^^^ value of type `GenericListArray` cannot be built from `std::iter::Iterator>>`
| |
| required by a bound introduced by this call
|
= help: the trait `FromIterator>>` is not implemented for `GenericListArray`, which is required by `Result<_, _>: FromIterator>, _>>`
= help: the trait `FromIterator>` is implemented for `Result`
= note: required for `Result, _>` to implement `FromIterator>, _>>`
note: required by a bound in `collect`
--> /Users/something/.rustup/toolchains/1.80-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2001:19
|
2001 | fn collect>(self) -> B
| ^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `Iterator::collect`
```

Contributor guide

Open the contributing guide

Research direction

Start with arrow-pg/src/lib.rs and the ListArray iterator and collection APIs, then run the shown list_array test to reproduce the FromIterator error. Done means establishing a validated approach for per-item filtering that preserves null lists and nested nulls.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.