[CH]CH upstream break the compatiability of ActionsDAG::split and it is not right.
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
CH (ClickHouse)
### Bug description
It will cause sqls that contains explode/posexplode crash. The issue was introduced in https://github.com/ClickHouse/ClickHouse/pull/58554
```
2024-02-20 16:31:26.288 ProjectRelParser: original actions_dag:0 : INPUT () (no column) Nullable(Int64) l_orderkey
1 : INPUT () (no column) Nullable(Int64) l_suppkey
2 : INPUT () (no column) Nullable(String) l_returnflag
3 : INPUT () (no column) Nullable(String) l_linestatus
4 : FUNCTION (2, 3) (no column) Array(Nullable(String)) array(l_returnflag,l_linestatus) [array]
5 : COLUMN () Const(Array(Nullable(String))) Array(Nullable(String)) []_0
6 : FUNCTION (4, 5) (no column) Array(Nullable(String)) ifNull(array(l_returnflag,l_linestatus),[]_0) [ifNull]
7 : FUNCTION (6) (no column) Array(Nullable(String)) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0)) [assumeNotNull]
8 : FUNCTION (7) (no column) Array(Nullable(String)) materialize(assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))) [materialize]
9 : ALIAS (8) (no column) Array(Nullable(String)) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
10 : ARRAY JOIN (9) (no column) Nullable(String) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
Output nodes: 0 1 10
Project input: 1
Projected output: 1
first split result: left:
0 : INPUT () (no column) Nullable(String) l_returnflag
1 : INPUT () (no column) Nullable(String) l_linestatus
2 : FUNCTION (0, 1) (no column) Array(Nullable(String)) array(l_returnflag,l_linestatus) [array]
3 : COLUMN () Const(Array(Nullable(String))) Array(Nullable(String)) []_0
4 : FUNCTION (2, 3) (no column) Array(Nullable(String)) ifNull(array(l_returnflag,l_linestatus),[]_0) [ifNull]
5 : FUNCTION (4) (no column) Array(Nullable(String)) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0)) [assumeNotNull]
6 : FUNCTION (5) (no column) Array(Nullable(String)) materialize(assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))) [materialize]
7 : ALIAS (6) (no column) Array(Nullable(String)) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
Output nodes: 7
Project input: 0
Projected output: 0
first split result: right:
0 : INPUT () (no column) Nullable(Int64) l_orderkey
1 : INPUT () (no column) Nullable(Int64) l_suppkey
2 : ARRAY JOIN (3) (no column) Nullable(String) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
3 : INPUT () (no column) Array(Nullable(String)) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
Output nodes: 0 1 2
Project input: 0
Projected output: 0
second split result: left:
0 : INPUT () (no column) Array(Nullable(String)) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
1 : ARRAY JOIN (0) (no column) Nullable(String) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
Output nodes: 1
Project input: 0
Projected output: 0
second split result: right:
0 : INPUT () (no column) Nullable(Int64) l_orderkey
1 : INPUT () (no column) Nullable(Int64) l_suppkey
2 : INPUT () (no column) Nullable(String) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
Output nodes: 0 1 2
Project input: 0
Projected output: 0
```
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Research direction
Start by reviewing ActionsDAG::split and the ClickHouse change linked in the issue, then reproduce a SQL query containing explode or posexplode. Done means those queries no longer crash and the split results preserve the required inputs and outputs shown in the logs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100