AbsaOSS / AbsaOSS/enceladus

Conformance doesn't allow using output field with lower level than the used column

Abierto
#759 1 comentario 0 reacciones 0 asignados Ver en GitHub
bug Conformance priority: medium
Lenguaje dominante
Scala
Estrellas
33
Forks
16
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

## Describe the bug
If I have dataset X with columns `a`, `b`, `c` and `d` on a root level, where `d` is a struct. I am unable to use Casting Conformance Rule on `a` with output column `d.aCasted`. This throws an error:
```bash
Exception in thread "main" za.co.absa.enceladus.conformance.interpreter.rules.ValidationException: X - Casting rule validation error: input and output columns don't have the same parent. Fields 'a' and 'd.aCasted' have different parents.
at za.co.absa.enceladus.conformance.interpreter.RuleValidators$.checkAndThrowValidationErrors(RuleValidators.scala:54)
at za.co.absa.enceladus.conformance.interpreter.RuleValidators$.validateSameParent(RuleValidators.scala:47)
at za.co.absa.enceladus.conformance.interpreter.rules.CastingRuleInterpreter.conform(CastingRuleInterpreter.scala:36)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.za$co$absa$enceladus$conformance$interpreter$DynamicInterpreter$$applyConformanceRule(DynamicInterpreter.scala:148)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$$anonfun$1.apply(DynamicInterpreter.scala:95)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$$anonfun$1.apply(DynamicInterpreter.scala:93)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.applyConformanceRules(DynamicInterpreter.scala:93)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.interpret(DynamicInterpreter.scala:64)
at za.co.absa.enceladus.conformance.DynamicConformanceJob$.conform(DynamicConformanceJob.scala:176)
at za.co.absa.enceladus.conformance.DynamicConformanceJob$.main(DynamicConformanceJob.scala:88)
at za.co.absa.enceladus.conformance.DynamicConformanceJob.main(DynamicConformanceJob.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```

## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Create Schema X with columns `a`, `b`, `c` and `d`, where `d` is a struct and other are Strings/Ints/Booleans/... whatever
2. Create Dataset X with schema X
3. Create a Casting Conformance rule where input is `a`, output field is `d.aCasted`, cast it to whatever.
4. Run Standardization and Conformance
5. Conformance throws an error

## Expected behaviour
I would expect to be able to use structures down the flow (on lower levels)

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.