Conformance doesn't allow using output field with lower level than the used column
- Ngôn ngữ chính
- Scala
- Star
- 33
- Fork
- 16
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
## Describe the bug
If I have dataset X with columns `a`, `b`, `c` and `d` on a root level, where `d` is a struct. I am unable to use Casting Conformance Rule on `a` with output column `d.aCasted`. This throws an error:
```bash
Exception in thread "main" za.co.absa.enceladus.conformance.interpreter.rules.ValidationException: X - Casting rule validation error: input and output columns don't have the same parent. Fields 'a' and 'd.aCasted' have different parents.
at za.co.absa.enceladus.conformance.interpreter.RuleValidators$.checkAndThrowValidationErrors(RuleValidators.scala:54)
at za.co.absa.enceladus.conformance.interpreter.RuleValidators$.validateSameParent(RuleValidators.scala:47)
at za.co.absa.enceladus.conformance.interpreter.rules.CastingRuleInterpreter.conform(CastingRuleInterpreter.scala:36)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.za$co$absa$enceladus$conformance$interpreter$DynamicInterpreter$$applyConformanceRule(DynamicInterpreter.scala:148)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$$anonfun$1.apply(DynamicInterpreter.scala:95)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$$anonfun$1.apply(DynamicInterpreter.scala:93)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.applyConformanceRules(DynamicInterpreter.scala:93)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.interpret(DynamicInterpreter.scala:64)
at za.co.absa.enceladus.conformance.DynamicConformanceJob$.conform(DynamicConformanceJob.scala:176)
at za.co.absa.enceladus.conformance.DynamicConformanceJob$.main(DynamicConformanceJob.scala:88)
at za.co.absa.enceladus.conformance.DynamicConformanceJob.main(DynamicConformanceJob.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Create Schema X with columns `a`, `b`, `c` and `d`, where `d` is a struct and other are Strings/Ints/Booleans/... whatever
2. Create Dataset X with schema X
3. Create a Casting Conformance rule where input is `a`, output field is `d.aCasted`, cast it to whatever.
4. Run Standardization and Conformance
5. Conformance throws an error
## Expected behaviour
I would expect to be able to use structures down the flow (on lower levels)
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.