apache / apache/gluten

[CH]CH upstream break the compatiability of ActionsDAG::split and it is not right.

Open
#4,723 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

CH (ClickHouse)

### Bug description

It will cause sqls that contains explode/posexplode crash. The issue was introduced in https://github.com/ClickHouse/ClickHouse/pull/58554

```
2024-02-20 16:31:26.288 ProjectRelParser: original actions_dag:0 : INPUT () (no column) Nullable(Int64) l_orderkey
1 : INPUT () (no column) Nullable(Int64) l_suppkey
2 : INPUT () (no column) Nullable(String) l_returnflag
3 : INPUT () (no column) Nullable(String) l_linestatus
4 : FUNCTION (2, 3) (no column) Array(Nullable(String)) array(l_returnflag,l_linestatus) [array]
5 : COLUMN () Const(Array(Nullable(String))) Array(Nullable(String)) []_0
6 : FUNCTION (4, 5) (no column) Array(Nullable(String)) ifNull(array(l_returnflag,l_linestatus),[]_0) [ifNull]
7 : FUNCTION (6) (no column) Array(Nullable(String)) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0)) [assumeNotNull]
8 : FUNCTION (7) (no column) Array(Nullable(String)) materialize(assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))) [materialize]
9 : ALIAS (8) (no column) Array(Nullable(String)) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
10 : ARRAY JOIN (9) (no column) Nullable(String) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
Output nodes: 0 1 10
Project input: 1
Projected output: 1

first split result: left:
0 : INPUT () (no column) Nullable(String) l_returnflag
1 : INPUT () (no column) Nullable(String) l_linestatus
2 : FUNCTION (0, 1) (no column) Array(Nullable(String)) array(l_returnflag,l_linestatus) [array]
3 : COLUMN () Const(Array(Nullable(String))) Array(Nullable(String)) []_0
4 : FUNCTION (2, 3) (no column) Array(Nullable(String)) ifNull(array(l_returnflag,l_linestatus),[]_0) [ifNull]
5 : FUNCTION (4) (no column) Array(Nullable(String)) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0)) [assumeNotNull]
6 : FUNCTION (5) (no column) Array(Nullable(String)) materialize(assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))) [materialize]
7 : ALIAS (6) (no column) Array(Nullable(String)) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
Output nodes: 7
Project input: 0
Projected output: 0
first split result: right:
0 : INPUT () (no column) Nullable(Int64) l_orderkey
1 : INPUT () (no column) Nullable(Int64) l_suppkey
2 : ARRAY JOIN (3) (no column) Nullable(String) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
3 : INPUT () (no column) Array(Nullable(String)) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
Output nodes: 0 1 2
Project input: 0
Projected output: 0

second split result: left:
0 : INPUT () (no column) Array(Nullable(String)) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
1 : ARRAY JOIN (0) (no column) Nullable(String) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
Output nodes: 1
Project input: 0
Projected output: 0

second split result: right:
0 : INPUT () (no column) Nullable(Int64) l_orderkey
1 : INPUT () (no column) Nullable(Int64) l_suppkey
2 : INPUT () (no column) Nullable(String) assumeNotNull(ifNull(array(l_returnflag,l_linestatus),[]_0))
Output nodes: 0 1 2
Project input: 0
Projected output: 0
```

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing ActionsDAG::split and the ClickHouse change linked in the issue, then reproduce a SQL query containing explode or posexplode. Done means those queries no longer crash and the split results preserve the required inputs and outputs shown in the logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.