AbsaOSS / AbsaOSS/enceladus

Add support for ignoring trailing line(s) in CSV

Ouverte
#1,324 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
feature priority: high request Standardization
Langage dominant
Scala
Étoiles
33
Forks
16
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

## Background
Sometimes input raw files contain a trailing line that needs to be ignored. We have a use case for the trailing 1 line, but if we need to implement this feature anyway, maybe let's consider do it more generally.

## Feature
Add support for a command-line option to ignore trailing lines of each input CSV file.

## Example [Optional]
```
a,b,c
1,2,3
5,6,7
Total=2
```

## Proposed Solution [Optional]
If there is no such option in `spark-csv`, possibly we need to preprocess incoming CSV files in RDDs.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.