AbsaOSS / AbsaOSS/enceladus

Non-trimmed numeric values can fail parsing in 'fixed-width' source

Ouverte
#1,411 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
bug priority: undecided Standardization
Langage dominant
Scala
Étoiles
33
Forks
16
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

## Describe the bug
This issue probably should be moved to `fixed-width` data source when it is created a separate repo. Adding it here so we won't forget.

`fixed-width` data source supports switching on/off trimming of values. When trimming is `true` leading and trailing spaces are removed from each value during reading. When if is `false` all leading and trailing spaces remain.

Turning off trimming leads to strange behavior when applied to numeric columns. If a number of digits in a number is less than the column width, the parsing fails and `null` is produced.

## To Reproduce
Examples from @Zejnilovic PR #1409

For data like this
```
1 Georgina Richardt
2 Seamus Klaes
3 Griff Leaning
4 Klarrisa Corps
5 Cornie Hewertson
```

We get
```
+---+---------------+---------+--------------------------------------------------------------------------+
|id |first_name |last_name|errCol |
+---+---------------+---------+--------------------------------------------------------------------------+
|0 |Georgina | Richardt|[[stdCastError, E00000, Standardization Error - Type cast, id, [1 ], []]]|
|0 |Seamus | Klaes|[[stdCastError, E00000, Standardization Error - Type cast, id, [2 ], []]]|
|0 |Griff | Leaning|[[stdCastError, E00000, Standardization Error - Type cast, id, [3 ], []]]|
|0 |Klarrisa | Corps|[[stdCastError, E00000, Standardization Error - Type cast, id, [4 ], []]]|
|0 |Cornie |Hewertson|[[stdCastError, E00000, Standardization Error - Type cast, id, [5 ], []]]|
+---+---------------+---------+--------------------------------------------------------------------------+
```

## Expected behaviour
When trimming is `false` we should still have numeric fields parsed correctly.
```
+---+---------------+---------+------+
|id |first_name |last_name|errCol|
+---+---------------+---------+------+
|1 |Georgina | Richardt|[] |
|2 |Seamus | Klaes|[] |
|3 |Griff | Leaning|[] |
|4 |Klarrisa | Corps|[] |
|5 |Cornie |Hewertson|[] |
+---+---------------+---------+------+
```

*e.g. Numeric fields need to be trimmed anyway before casting*

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.