AbsaOSS / AbsaOSS/enceladus

When updating schema on dataset there is no check for collisions with conformance rules

Open
#715 2 comments 1 reaction 0 assignees View on GitHub
bug Menas priority: medium
Dominant language
Scala
Stars
33
Forks
16
PR merge metrics
No merged PRs in 30d

Description

## Describe the bug
If I have a schema with columns `a`,`b`,`c`, create a column `d` (string) using conformance rule and then update schema to contain `d` (timestamp) column of different kind and change it for the dataset, there is nothing stopping me and the mistake will be only evident in the middle of conformance when it tells me there already is column `d`.

## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Have dataset `X` with columns `a`,`b`,`c`
2. Add conformance rule that creates columns `d`
3. Change schema of dataset `X` to have column `d` from raw.
4. Run standardization and conformance
5. See error from conformance in console/logger/STDOUT

## Expected behaviour
When changing/updating schema check if it is possible depending on output columns of conformance rules.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.