apache / apache/arrow

[C++][Acero] Add support for `ListType` for non-key fields to Hash Joins

Open
#43,716 13 comments 6 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Acero's Hash Join does not support `ListType` in non-key fields for a hash join: https://github.com/apache/arrow/blob/main/cpp/src/arrow/acero/hash_join_node.cc#L48 . This is a request to add that support.

PyArrow code that reproduces here:

```
import pyarrow as pa
import pyarrow.acero as acero

# Creating the Arrow tables
basic_tbl = pa.table({'x': [1, 2, 3], 'y': ['a', 'b', 'c']})
basic_tbl_src = acero.Declaration("table_source", options=acero.TableSourceNodeOptions(basic_tbl))

basic_tbl2 = pa.table({'x': [1, 2, 3], 'z': [True, False, True]})
basic_tbl2_src = acero.Declaration("table_source", options=acero.TableSourceNodeOptions(basic_tbl2))

list_tbl = pa.table({'z': [['first', 'list', 'col', 'row'], ['second row', 'here']], 'x': [1, 2]})
list_tbl_src = acero.Declaration("table_source", options=acero.TableSourceNodeOptions(list_tbl))

join_keys = ["x"]

hash_join_options = acero.HashJoinNodeOptions('left outer', left_keys=join_keys, right_keys=join_keys)

joined = acero.Declaration(
"hashjoin", options=hash_join_options, inputs=[basic_tbl_src, basic_tbl2_src])

result = joined.to_table()
print(result)

# list table
joined = acero.Declaration(
"hashjoin", options=hash_join_options, inputs=[basic_tbl_src, list_tbl_src])

result = joined.to_table()
print(result)

```

R code here: https://issues.apache.org/jira/browse/ARROW-14519

In [that link](https://issues.apache.org/jira/browse/ARROW-14519), the reason there currently isn't support was noted:

> We cannot easily support more types in hash join right now. That is because we transform and encode all the input values, key and non-key (row_encoder.h), so it would need another specialization for each additional type.

So to add this support, it seems like we will need to add the specialisation for the encoding of `ListType`.

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.