AbsaOSS / AbsaOSS/enceladus

Defensively use crossJoin syntax rather than setting crossJoin.enabled to true

未关闭
#895 4 条评论 0 个 reaction 已指派 1 人 已被 @yruslan 认领 在 GitHub 查看
Conformance feature priority: high under discussion
主要语言
Scala
星标
33
派生
16
PR 合并指标
30 天内没有已合并 PR

描述

As a fix for issue #892 we configure `spark.sql.crossJoin.enabled` to `true` in both Mapping Rule interpreters.

This configuration then remains true for the remainder of the application life cycle.

This is quite a dangerous thing to do as it may hide future bugs and warnings, which would otherwise raise AnalysisException in tests etc.

Much better approach would be to:
- investigate this bug a little further
- it's currently assumed, that this issue only arises when there's a single join condition using a column previously created by the literal rule
- it's only reproducible locally when the data and mapping tables are read from files
- we should analyse the list of conformance rules and in cases, where there's a mapping rule with a single join condition (one of which was previously created by literal) we should use the crossJoin syntax explicitly rather than left_outer join

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。