AbsaOSS / AbsaOSS/enceladus

Add validation that output column should be at the same array level

Open
#1,296 0 comments 0 reactions 0 assignees View on GitHub
bug Conformance priority: medium
Dominant language
Scala
Stars
33
Forks
16
PR merge metrics
No merged PRs in 30d

Description

## Describe the bug
For a schema like this:
```
root
|-- id: long (nullable = true)
|-- array1: array (nullable = true)
| |-- element: struct (containsNull = false)
| | |-- key1: long (nullable = true)
| | |-- key2: long (nullable = true)
```
if a join condition for a mapping rule includes `key1` and.or `key2`, then the output column should be inside `array1`. It cannot be at root level. Otherwise the error will occur:

```
Caused by: org.apache.spark.sql.AnalysisException: cannot resolve 'UDF(array1.key1, array1.key2)' due to data type mismatch: argument 2 requires long type, however, 'array1.key' is of array type.;;
```

## Expected behaviour
This case should be validated by Conformance before actually executing a mapping rule.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.