apache / apache/gluten

[CH] Incorrect result when using df.queryExecution.executeCollect directly

Open
#9,343 2 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

CH (ClickHouse)

### Bug description

```
def join(hint: String): DataFrame = {
sql(
s"""
|SELECT /*+ $hint */ *
|FROM testData t1
|LEFT OUTER JOIN
|testData2 t2
|ON key = a AND concat(value, b) = '12'
|""".stripMargin)
}

val df1 = join("SHUFFLE_HASH(t1)")
val res1 = df1.collect()
val res2 = df1.queryExecution.executedPlan.executeCollectPublic()
println(res1.size)
println(res2.size)
```

res1.size = 100, res2.size = 0

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start with the Scala reproducer in the issue and compare df1.collect() with df1.queryExecution.executedPlan.executeCollectPublic() for the ClickHouse backend. Investigate the relevant execution path and establish a test using the provided join and hint; done means both collection paths return the same result size.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, scala
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.