AbsaOSS / AbsaOSS/enceladus

Conformance doesn't allow using output field with lower level than the used column

Open
#759 1 comment 0 reactions 0 assignees View on GitHub
bug Conformance priority: medium
Dominant language
Scala
Stars
33
Forks
16
PR merge metrics
No merged PRs in 30d

Description

## Describe the bug
If I have dataset X with columns `a`, `b`, `c` and `d` on a root level, where `d` is a struct. I am unable to use Casting Conformance Rule on `a` with output column `d.aCasted`. This throws an error:
```bash
Exception in thread "main" za.co.absa.enceladus.conformance.interpreter.rules.ValidationException: X - Casting rule validation error: input and output columns don't have the same parent. Fields 'a' and 'd.aCasted' have different parents.
at za.co.absa.enceladus.conformance.interpreter.RuleValidators$.checkAndThrowValidationErrors(RuleValidators.scala:54)
at za.co.absa.enceladus.conformance.interpreter.RuleValidators$.validateSameParent(RuleValidators.scala:47)
at za.co.absa.enceladus.conformance.interpreter.rules.CastingRuleInterpreter.conform(CastingRuleInterpreter.scala:36)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.za$co$absa$enceladus$conformance$interpreter$DynamicInterpreter$$applyConformanceRule(DynamicInterpreter.scala:148)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$$anonfun$1.apply(DynamicInterpreter.scala:95)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$$anonfun$1.apply(DynamicInterpreter.scala:93)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.applyConformanceRules(DynamicInterpreter.scala:93)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.interpret(DynamicInterpreter.scala:64)
at za.co.absa.enceladus.conformance.DynamicConformanceJob$.conform(DynamicConformanceJob.scala:176)
at za.co.absa.enceladus.conformance.DynamicConformanceJob$.main(DynamicConformanceJob.scala:88)
at za.co.absa.enceladus.conformance.DynamicConformanceJob.main(DynamicConformanceJob.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```

## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Create Schema X with columns `a`, `b`, `c` and `d`, where `d` is a struct and other are Strings/Ints/Booleans/... whatever
2. Create Dataset X with schema X
3. Create a Casting Conformance rule where input is `a`, output field is `d.aCasted`, cast it to whatever.
4. Run Standardization and Conformance
5. Conformance throws an error

## Expected behaviour
I would expect to be able to use structures down the flow (on lower levels)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.