AbsaOSS / AbsaOSS/enceladus

Conformance doesn't allow using output field with lower level than the used column

Offen
#759 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug Conformance priority: medium
Vorherrschende Sprache
Scala
Sterne
33
Forks
16
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## Describe the bug
If I have dataset X with columns `a`, `b`, `c` and `d` on a root level, where `d` is a struct. I am unable to use Casting Conformance Rule on `a` with output column `d.aCasted`. This throws an error:
```bash
Exception in thread "main" za.co.absa.enceladus.conformance.interpreter.rules.ValidationException: X - Casting rule validation error: input and output columns don't have the same parent. Fields 'a' and 'd.aCasted' have different parents.
at za.co.absa.enceladus.conformance.interpreter.RuleValidators$.checkAndThrowValidationErrors(RuleValidators.scala:54)
at za.co.absa.enceladus.conformance.interpreter.RuleValidators$.validateSameParent(RuleValidators.scala:47)
at za.co.absa.enceladus.conformance.interpreter.rules.CastingRuleInterpreter.conform(CastingRuleInterpreter.scala:36)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.za$co$absa$enceladus$conformance$interpreter$DynamicInterpreter$$applyConformanceRule(DynamicInterpreter.scala:148)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$$anonfun$1.apply(DynamicInterpreter.scala:95)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$$anonfun$1.apply(DynamicInterpreter.scala:93)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.applyConformanceRules(DynamicInterpreter.scala:93)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.interpret(DynamicInterpreter.scala:64)
at za.co.absa.enceladus.conformance.DynamicConformanceJob$.conform(DynamicConformanceJob.scala:176)
at za.co.absa.enceladus.conformance.DynamicConformanceJob$.main(DynamicConformanceJob.scala:88)
at za.co.absa.enceladus.conformance.DynamicConformanceJob.main(DynamicConformanceJob.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```

## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Create Schema X with columns `a`, `b`, `c` and `d`, where `d` is a struct and other are Strings/Ints/Booleans/... whatever
2. Create Dataset X with schema X
3. Create a Casting Conformance rule where input is `a`, output field is `d.aCasted`, cast it to whatever.
4. Run Standardization and Conformance
5. Conformance throws an error

## Expected behaviour
I would expect to be able to use structures down the flow (on lower levels)

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.