apache / apache/datafusion

Fuse operations in `equal_rows_arr`

Open
#12,131 4 comments 0 reactions 1 assignee Claimed by @LeslieKid View on GitHub
enhancement performance
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

equal_rows_arr compares pairs of 2 arrays with indices for equality but shows up in profiles.

Currently this is done in the following way

* `take` the values for the indices for the first pair
* comparing the arrays using `eq` or `not_distinct`
* doing the same for the next pairs and `and`ing the results
* Filtering the indices based on the resulting boolean array

### Describe the solution you'd like

We could optimize this in some ways:

* Writing a kernel that doesn't use `take` (i.e. copy the array) but compares arrays based on the indices.
* Target creating a single `RecordBatch` using `BatchCoalescer` rather than creating multiple small batches and concatenating afterwards
* Writing results to a single booleanbuffer rather than creating a new one every time
* Removing indices from the list (e.g. using `Vec::retain`) not matching rather than creating a boolean array for a filter
* Reuse allocations as much as possible between batches

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.