AbsaOSS / AbsaOSS/enceladus

Defensively use crossJoin syntax rather than setting crossJoin.enabled to true

Đang mở
#895 4 bình luận 0 reaction 1 người được giao Được @yruslan nhận Xem trên GitHub
Conformance feature priority: high under discussion
Ngôn ngữ chính
Scala
Star
33
Fork
16
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

As a fix for issue #892 we configure `spark.sql.crossJoin.enabled` to `true` in both Mapping Rule interpreters.

This configuration then remains true for the remainder of the application life cycle.

This is quite a dangerous thing to do as it may hide future bugs and warnings, which would otherwise raise AnalysisException in tests etc.

Much better approach would be to:
- investigate this bug a little further
- it's currently assumed, that this issue only arises when there's a single join condition using a column previously created by the literal rule
- it's only reproducible locally when the data and mapping tables are read from files
- we should analyse the list of conformance rules and in cases, where there's a mapping rule with a single join condition (one of which was previously created by literal) we should use the crossJoin syntax explicitly rather than left_outer join

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.