AbsaOSS / AbsaOSS/enceladus

Defensively use crossJoin syntax rather than setting crossJoin.enabled to true

Abierto
#895 4 comentarios 0 reacciones 1 asignado Reclamado por @yruslan Ver en GitHub
Conformance feature priority: high under discussion
Lenguaje dominante
Scala
Estrellas
33
Forks
16
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

As a fix for issue #892 we configure `spark.sql.crossJoin.enabled` to `true` in both Mapping Rule interpreters.

This configuration then remains true for the remainder of the application life cycle.

This is quite a dangerous thing to do as it may hide future bugs and warnings, which would otherwise raise AnalysisException in tests etc.

Much better approach would be to:
- investigate this bug a little further
- it's currently assumed, that this issue only arises when there's a single join condition using a column previously created by the literal rule
- it's only reproducible locally when the data and mapping tables are read from files
- we should analyse the list of conformance rules and in cases, where there's a mapping rule with a single join condition (one of which was previously created by literal) we should use the crossJoin syntax explicitly rather than left_outer join

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.