AbsaOSS / AbsaOSS/spline-spark-agent

UNION should propagate identical input attributes to its output

未关闭
#270 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Scala
星标
210
派生
102
平均合并
1 天 1 小时
30 天内合并 PR
1

描述

Consider the following case:
```
(a, b, c)
/ \
. .
SELECT a= .
. .
. SELECT b=
. .
\ /
UNION
|
(a', b', c')
```

The `UNION` combines two data frames that are created from the same shared data frame `(a, b, c)` by applying different transformation chains.

To preserve both left and right paths in the resulting `(a', b', c')` data frame lineage we create new synthetic attributes for the `UNION` output, and connecting those attributes to all corresponding attributes in the input data frames.

The problem arises when some of those attributes are identical, e.g. in the above example, attribute `c` isn't touched and is simply propagated everywhere. Thus in the resulting data frame the attribute `c` is expected to be presented itself, not `c' = f(c)` as Spline currently shows.

Technically speaking `c' = f(c)` is not particularly incorrect representation, as `f` could be an identity function so the expression boils down to just the `c' = c` as expected. But it just creates unnecessary clutter in the graph structure with duplicated intermediate attribute names making it difficult to reason about.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。