apache / apache/datafusion

Refactoring of array_set operations (array_union, array_except, array_intersection)

Open
#8,156 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

All these functions require fundamentally traversing row-by-row two arrays and creating a row-by-row result, so it would be nice if we could have a single implementation of the traversal and use a trait to inject the custom behavior

### Describe the solution you'd like

We could have something like this:

```rust
trait ArraySetOperationAccumulator<'a> {
fn new() -> Self;
fn process_left(&mut self, row: Row<'a>) -> Result<()>;
fn process_right(&mut self, row: Row<'a>) -> Result<()>;
fn rows(&self) -> &[Row<'a>];
fn clear(&mut self) -> ();
}
```

one of the problem of this implementation is the lifetime of Rows, but we can maybe work around it if we use a .clone() on the rows and remove the lifetime

### Describe alternatives you've considered

Duplicating the traversing code for each of these function, as it is already happening

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the current array_union, array_except, and array_intersection implementations and compare how each traverses rows. Examine the Row lifetime constraints and the proposed ArraySetOperationAccumulator shape; done means the operations share traversal logic without changing their behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.