Fuse operations in `equal_rows_arr`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
equal_rows_arr compares pairs of 2 arrays with indices for equality but shows up in profiles.
Currently this is done in the following way
* `take` the values for the indices for the first pair
* comparing the arrays using `eq` or `not_distinct`
* doing the same for the next pairs and `and`ing the results
* Filtering the indices based on the resulting boolean array
### Describe the solution you'd like
We could optimize this in some ways:
* Writing a kernel that doesn't use `take` (i.e. copy the array) but compares arrays based on the indices.
* Target creating a single `RecordBatch` using `BatchCoalescer` rather than creating multiple small batches and concatenating afterwards
* Writing results to a single booleanbuffer rather than creating a new one every time
* Removing indices from the list (e.g. using `Vec::retain`) not matching rather than creating a boolean array for a filter
* Reuse allocations as much as possible between batches
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.