AbsaOSS / AbsaOSS/enceladus

Non-trimmed numeric values can fail parsing in 'fixed-width' source

Offen
#1,411 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug priority: undecided Standardization
Vorherrschende Sprache
Scala
Sterne
33
Forks
16
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## Describe the bug
This issue probably should be moved to `fixed-width` data source when it is created a separate repo. Adding it here so we won't forget.

`fixed-width` data source supports switching on/off trimming of values. When trimming is `true` leading and trailing spaces are removed from each value during reading. When if is `false` all leading and trailing spaces remain.

Turning off trimming leads to strange behavior when applied to numeric columns. If a number of digits in a number is less than the column width, the parsing fails and `null` is produced.

## To Reproduce
Examples from @Zejnilovic PR #1409

For data like this
```
1 Georgina Richardt
2 Seamus Klaes
3 Griff Leaning
4 Klarrisa Corps
5 Cornie Hewertson
```

We get
```
+---+---------------+---------+--------------------------------------------------------------------------+
|id |first_name |last_name|errCol |
+---+---------------+---------+--------------------------------------------------------------------------+
|0 |Georgina | Richardt|[[stdCastError, E00000, Standardization Error - Type cast, id, [1 ], []]]|
|0 |Seamus | Klaes|[[stdCastError, E00000, Standardization Error - Type cast, id, [2 ], []]]|
|0 |Griff | Leaning|[[stdCastError, E00000, Standardization Error - Type cast, id, [3 ], []]]|
|0 |Klarrisa | Corps|[[stdCastError, E00000, Standardization Error - Type cast, id, [4 ], []]]|
|0 |Cornie |Hewertson|[[stdCastError, E00000, Standardization Error - Type cast, id, [5 ], []]]|
+---+---------------+---------+--------------------------------------------------------------------------+
```

## Expected behaviour
When trimming is `false` we should still have numeric fields parsed correctly.
```
+---+---------------+---------+------+
|id |first_name |last_name|errCol|
+---+---------------+---------+------+
|1 |Georgina | Richardt|[] |
|2 |Seamus | Klaes|[] |
|3 |Griff | Leaning|[] |
|4 |Klarrisa | Corps|[] |
|5 |Cornie |Hewertson|[] |
+---+---------------+---------+------+
```

*e.g. Numeric fields need to be trimmed anyway before casting*

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.