AbsaOSS / AbsaOSS/enceladus

New version creation of a dataset do not carry over the properties

Offen
#1,698 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug priority: undecided
Vorherrschende Sprache
Scala
Sterne
33
Forks
16
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## Describe the bug
Having a testing required properties key on my local Menas and having such a key-value pair in the properties of a dataset I want to run, I have discovered that if a new version of the dataset is created (e.g. by adding description, adding a conformance rule, ...) the properties set to the previous version do not carry over.

This, in turn, may prevent running standardization/conformance if the system has any required properties keys set.

## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Have a dataset name `ds1` in the latest version `7` that works for std/conf
2. Update your system to require a property `requiredKey1`:
`POST http://localhost:8080/menas/api/properties/datasets` with a payload
```json
{
"name": "requiredKey1",
"description": "myDesc",
"propertyType": {
"_t": "StringPropertyType",
"suggestedValue": "sortOfDefault"
},
"essentiality": {
"_t": "Mandatory"
}
}
```
3. This renders your dataset `ds1` unusable because it lacks the required key-value in its properties
4. Fix the dataset by providing the required key-value by
`PUT http://localhost:8080/menas/api/dataset/ds1/properties` with a payload
```json
{
"requiredKey1": "valueOfRequiredKey1"
}
```
resulting in version 8 of `ds1` that is usable for std/conf
5. Edit the dataset `ds1` by changing its description resulting in version 9.
6. Dataset `ds1` in version 9, now, has no properties set and is unusable for std/conf processing again (spark-submit will fail with this message)

## Expected behavior
It would be expected that the properties would carry over from v 8 to v 9.

## Comments
This is quite serious in cases where we expect that there may be any properties set in the system, until such time, the problem should not be encountered. That being said, it should be probably fixed sooner than later.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.