/operations/{operationId} output doesn't contain textual transformation expressions
- Lenguaje dominante
- Scala
- Estrellas
- 667
- Forks
- 155
- Merge medio
- 3 d 6 h
- PR fusionados (30 d)
- 4
Descripción
## Background
When a field in a project list is created or transformed in some fashion, there tends to be some sort of an original "expression" string that represents the transformation. For example, here is a simple field that gets created from two existing fields via a concat expression (Java sample)
df = df.withColumn(
"name",
concat(df.col("lname"), lit(", "), df.col("fname")));
So basically "name" is derived from "lname" and "fname" via some transformation. The attribute lineage itself is easy enough to figure out by analyzing the result of the /operations/{operationId} REST call.
However, it's not so easy to get back the human readable "text" of the transformation.
## Example
In the JSON returned by the /operations/{operationId}, I see something like
...
"projectList": [
{
"_typeHint": "expr.AttrRef",
"refId": "b9f83e1b-abe9-4555-b6a0-1ca0f6f23d50",
"expression" = "lname" //very useful, but not available
},
{
"_typeHint": "expr.AttrRef",
"refId": "a32219e0-eab0-4f0e-95de-d91ce2f43744",
"expression" = "fname" //very useful, but not available
},
{
"_typeHint": "expr.Alias",
"alias": "name",
"expression" : "concat(df.col("lname"), lit(", "), df.col("fname"))", //very useful, but not available
"child": {
"children": [
{
"_typeHint": "expr.AttrRef",
"refId": "b9f83e1b-abe9-4555-b6a0-1ca0f6f23d50"
},
{
"_typeHint": "expr.Literal",
"value": ", ",
"dataTypeId": "92c3dfe1-e004-4ca7-976c-a15a05a7da59"
},
{
"_typeHint": "expr.AttrRef",
"refId": "a32219e0-eab0-4f0e-95de-d91ce2f43744"
}
],
"name": "concat",
"_typeHint": "expr.Generic",
"dataTypeId": "28bdd23a-5004-421a-9b36-14458dec80a0",
"exprType": "Concat"
}
}
],
...
## Proposed Solution
Solution Ideas
As you can see, the original expression concat(df.col("lname"), lit(", "), df.col("fname")) is only represented in the REST JSON by its complex, structured version, which is not suitable for display to the user viewing the lineage. There is nothing else in the JSON I could find that has a textual version of the field-level transformation.
P.S. Note that in the Web UI for spline - you already seem to do something along these lines by showing an expression.
(Shown in Spline UI - but doesn't appear to be there in REST API output)
Transformations
λ = lname
λ = fname
λ = concatlname, , , fname AS name
`
So why not make the textual expressions above available via the REST API itself as that makes it useful for building a custom UI based purely on the REST API. I imagine adding an "expression" attribute (see my JSON snippet above) or similar would do the trick. "concatlname, , , fname AS name" above is obviously useful, but ideally having the original "concat(df.col("lname"), lit(", "), df.col("fname"))" is ideal if available (or even both versions)
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.