Excessive Arc-clone in HashJoinStream with StringView on build-side
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
An unfortunate pattern in the hash join implementation leads to excessive Arc-cloning: Assume the build-side carries a string-view column as a payload. Let N be the number of batches seen on the build side
1. In the build phase, datafusion concatenates the batches on the build side. The string-view column now holds references to at least N data buffers in a vec;
2. When constructing the output batch, the `take` implementation for string-views clones the data buffer vector of the concatenated build-side column - thus incrementing the references on all N data buffers.
### To Reproduce
I noticed this issue when executing and profiling tpch query 18 - roughly 3% of the runtime is spent cloning these Arcs.
### Expected behavior
_No response_
### Additional context
- The concat during build:
https://github.com/apache/datafusion/blob/7002a0027876a17e5bdf275e63d2a25373331943/datafusion/physical-plan/src/joins/hash_join.rs#L1013-L1015
- The take call during batch construction:
https://github.com/apache/datafusion/blob/7002a0027876a17e5bdf275e63d2a25373331943/datafusion/physical-plan/src/joins/utils.rs#L918
- The relevant bit of arrow-rs
https://github.com/apache/arrow-rs/blob/7e85b48dc8f929afa82f2878b17db7b2df240b8b/arrow-select/src/take.rs#L565-L567
Contributor guide
Assessment
This issue has not been assessed yet.