AbsaOSS / AbsaOSS/spline

/operations/{operationId} output doesn't contain textual transformation expressions

Open
#801 6 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Scala
Stars
667
Forks
155
Avg merge
3d 6h
Merged PRs (30d)
4

Description

## Background

When a field in a project list is created or transformed in some fashion, there tends to be some sort of an original "expression" string that represents the transformation. For example, here is a simple field that gets created from two existing fields via a concat expression (Java sample)

df = df.withColumn(
"name",
concat(df.col("lname"), lit(", "), df.col("fname")));

So basically "name" is derived from "lname" and "fname" via some transformation. The attribute lineage itself is easy enough to figure out by analyzing the result of the /operations/{operationId} REST call.

However, it's not so easy to get back the human readable "text" of the transformation.

## Example
In the JSON returned by the /operations/{operationId}, I see something like

...
"projectList": [
{
"_typeHint": "expr.AttrRef",
"refId": "b9f83e1b-abe9-4555-b6a0-1ca0f6f23d50",
"expression" = "lname" //very useful, but not available
},
{
"_typeHint": "expr.AttrRef",
"refId": "a32219e0-eab0-4f0e-95de-d91ce2f43744",
"expression" = "fname" //very useful, but not available
},
{
"_typeHint": "expr.Alias",
"alias": "name",
"expression" : "concat(df.col("lname"), lit(", "), df.col("fname"))", //very useful, but not available
"child": {
"children": [
{
"_typeHint": "expr.AttrRef",
"refId": "b9f83e1b-abe9-4555-b6a0-1ca0f6f23d50"
},
{
"_typeHint": "expr.Literal",
"value": ", ",
"dataTypeId": "92c3dfe1-e004-4ca7-976c-a15a05a7da59"
},
{
"_typeHint": "expr.AttrRef",
"refId": "a32219e0-eab0-4f0e-95de-d91ce2f43744"
}
],
"name": "concat",
"_typeHint": "expr.Generic",
"dataTypeId": "28bdd23a-5004-421a-9b36-14458dec80a0",
"exprType": "Concat"
}
}
],
...

## Proposed Solution
Solution Ideas

As you can see, the original expression concat(df.col("lname"), lit(", "), df.col("fname")) is only represented in the REST JSON by its complex, structured version, which is not suitable for display to the user viewing the lineage. There is nothing else in the JSON I could find that has a textual version of the field-level transformation.

P.S. Note that in the Web UI for spline - you already seem to do something along these lines by showing an expression.

(Shown in Spline UI - but doesn't appear to be there in REST API output)

Transformations
λ = lname
λ = fname
λ = concatlname, , , fname AS name
`
So why not make the textual expressions above available via the REST API itself as that makes it useful for building a custom UI based purely on the REST API. I imagine adding an "expression" attribute (see my JSON snippet above) or similar would do the trick. "concatlname, , , fname AS name" above is obviously useful, but ideally having the original "concat(df.col("lname"), lit(", "), df.col("fname"))" is ideal if available (or even both versions)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.