[BUG] AST join slows down significantly with a small left table
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
We found an issue late in 21.10 (https://github.com/NVIDIA/spark-rapids/issues/3736) with the AST driven inner join where given a smaller left side table our performance would drop significantly with AST, as opposed to using an unconditional join + a separate filter. This issue is to discuss whether cuDF could detect this and adjust how it handles such a join.
If the larger table is on the left, AST is at least ~1.3x faster than join + filter approach. If the smaller table is on the left, AST can be 10x slower than the join + filter approach. We can work around this for inner joins specifically, since we can swap in the plugin the sides fairly easily (https://github.com/NVIDIA/spark-rapids/issues/3832), but this is hacky and seems like it is something cuDF should be able to handle, especially for all the other types of joins where we can't just swap sides like this.
The reason for the swap of tables in Spark, is that there is logic to build left or build right, depending on the byte sizes of the inputs. The spark-rapids plugin will split into smaller chunks the left side (if building right), or the right side (if building left), streaming these chunks against the side of the join that wasn't split (aka the build side). In this case we went from a build-left regime which was good for the AST since the left side was not split, to a build-right regime when the left side started to get split.
The reason why these kernels are so sensitive to the left side, is because that's what is getting used to compute the grid size:
```
detail::grid_1d config(left_table->num_rows(), DEFAULT_JOIN_BLOCK_SIZE);
```
And in our example (https://github.com/NVIDIA/spark-rapids/issues/3736) the left table could be hundreds of rows, where the right table could be 1M rows.
Contributor guide
Assessment
This issue has not been assessed yet.