AbsaOSS / AbsaOSS/enceladus

Conformance doesn't allow using output field with lower level than the used column

Đang mở
#759 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
bug Conformance priority: medium
Ngôn ngữ chính
Scala
Star
33
Fork
16
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

## Describe the bug
If I have dataset X with columns `a`, `b`, `c` and `d` on a root level, where `d` is a struct. I am unable to use Casting Conformance Rule on `a` with output column `d.aCasted`. This throws an error:
```bash
Exception in thread "main" za.co.absa.enceladus.conformance.interpreter.rules.ValidationException: X - Casting rule validation error: input and output columns don't have the same parent. Fields 'a' and 'd.aCasted' have different parents.
at za.co.absa.enceladus.conformance.interpreter.RuleValidators$.checkAndThrowValidationErrors(RuleValidators.scala:54)
at za.co.absa.enceladus.conformance.interpreter.RuleValidators$.validateSameParent(RuleValidators.scala:47)
at za.co.absa.enceladus.conformance.interpreter.rules.CastingRuleInterpreter.conform(CastingRuleInterpreter.scala:36)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.za$co$absa$enceladus$conformance$interpreter$DynamicInterpreter$$applyConformanceRule(DynamicInterpreter.scala:148)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$$anonfun$1.apply(DynamicInterpreter.scala:95)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$$anonfun$1.apply(DynamicInterpreter.scala:93)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.applyConformanceRules(DynamicInterpreter.scala:93)
at za.co.absa.enceladus.conformance.interpreter.DynamicInterpreter$.interpret(DynamicInterpreter.scala:64)
at za.co.absa.enceladus.conformance.DynamicConformanceJob$.conform(DynamicConformanceJob.scala:176)
at za.co.absa.enceladus.conformance.DynamicConformanceJob$.main(DynamicConformanceJob.scala:88)
at za.co.absa.enceladus.conformance.DynamicConformanceJob.main(DynamicConformanceJob.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```

## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Create Schema X with columns `a`, `b`, `c` and `d`, where `d` is a struct and other are Strings/Ints/Booleans/... whatever
2. Create Dataset X with schema X
3. Create a Casting Conformance rule where input is `a`, output field is `d.aCasted`, cast it to whatever.
4. Run Standardization and Conformance
5. Conformance throws an error

## Expected behaviour
I would expect to be able to use structures down the flow (on lower levels)

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.