AbsaOSS / AbsaOSS/spline-spark-agent

UNION should propagate identical input attributes to its output

Aperta
#270 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Scala
Stelle
210
Fork
102
Merge medio
1g 1h
PR unite (30g)
1

Descrizione

Consider the following case:
```
(a, b, c)
/ \
. .
SELECT a= .
. .
. SELECT b=
. .
\ /
UNION
|
(a', b', c')
```

The `UNION` combines two data frames that are created from the same shared data frame `(a, b, c)` by applying different transformation chains.

To preserve both left and right paths in the resulting `(a', b', c')` data frame lineage we create new synthetic attributes for the `UNION` output, and connecting those attributes to all corresponding attributes in the input data frames.

The problem arises when some of those attributes are identical, e.g. in the above example, attribute `c` isn't touched and is simply propagated everywhere. Thus in the resulting data frame the attribute `c` is expected to be presented itself, not `c' = f(c)` as Spline currently shows.

Technically speaking `c' = f(c)` is not particularly incorrect representation, as `f` could be an identity function so the expression boils down to just the `c' = c` as expected. But it just creates unnecessary clutter in the graph structure with duplicated intermediate attribute names making it difficult to reason about.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.