AbsaOSS / AbsaOSS/enceladus

Catalyst optimizer throws an 'implicit cross join' exception when joining on a literal

未关闭
#947 0 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
3rd party issue bug Conformance migration
主要语言
Scala
星标
33
派生
16
PR 合并指标
30 天内没有已合并 PR

描述

## Describe the bug
The issue #895 is related to a bug in Catalyst optimizer resulting in the creation of an implicit cross join. This bug is known and there are 2 Spark jiras raised about this already.

## To Reproduce
The shortest Spark job to replicate the issue is described in [SPARK-29176](https://issues.apache.org/jira/browse/SPARK-29176):
```scala
case class Value(id: Int, lower: String, upper: String)

import spark.implicits._

val values = Seq(Value(1, "one", "ONE")).toDS
val join = values.join(values.withColumn("id", lit(1)), "id")
```

## Expected behaviour
If the original execution plan passes the implicit cross join test so should the optimized plan.

## Spark versions affected
- Spark 2.2.2 - no issue
- Spark 2.4.3, 2.4.4, master - the issue occurs

## Spark JIRAs to track
- [SPARK-29176](https://issues.apache.org/jira/browse/SPARK-29176)
- [SPARK-24839](https://issues.apache.org/jira/browse/SPARK-24839)

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。