AbsaOSS / AbsaOSS/enceladus

Non-trimmed numeric values can fail parsing in 'fixed-width' source

Aperta
#1,411 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug priority: undecided Standardization
Lingua principale
Scala
Stelle
33
Fork
16
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Describe the bug
This issue probably should be moved to `fixed-width` data source when it is created a separate repo. Adding it here so we won't forget.

`fixed-width` data source supports switching on/off trimming of values. When trimming is `true` leading and trailing spaces are removed from each value during reading. When if is `false` all leading and trailing spaces remain.

Turning off trimming leads to strange behavior when applied to numeric columns. If a number of digits in a number is less than the column width, the parsing fails and `null` is produced.

## To Reproduce
Examples from @Zejnilovic PR #1409

For data like this
```
1 Georgina Richardt
2 Seamus Klaes
3 Griff Leaning
4 Klarrisa Corps
5 Cornie Hewertson
```

We get
```
+---+---------------+---------+--------------------------------------------------------------------------+
|id |first_name |last_name|errCol |
+---+---------------+---------+--------------------------------------------------------------------------+
|0 |Georgina | Richardt|[[stdCastError, E00000, Standardization Error - Type cast, id, [1 ], []]]|
|0 |Seamus | Klaes|[[stdCastError, E00000, Standardization Error - Type cast, id, [2 ], []]]|
|0 |Griff | Leaning|[[stdCastError, E00000, Standardization Error - Type cast, id, [3 ], []]]|
|0 |Klarrisa | Corps|[[stdCastError, E00000, Standardization Error - Type cast, id, [4 ], []]]|
|0 |Cornie |Hewertson|[[stdCastError, E00000, Standardization Error - Type cast, id, [5 ], []]]|
+---+---------------+---------+--------------------------------------------------------------------------+
```

## Expected behaviour
When trimming is `false` we should still have numeric fields parsed correctly.
```
+---+---------------+---------+------+
|id |first_name |last_name|errCol|
+---+---------------+---------+------+
|1 |Georgina | Richardt|[] |
|2 |Seamus | Klaes|[] |
|3 |Griff | Leaning|[] |
|4 |Klarrisa | Corps|[] |
|5 |Cornie |Hewertson|[] |
+---+---------------+---------+------+
```

*e.g. Numeric fields need to be trimmed anyway before casting*

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.