AbsaOSS / AbsaOSS/fixed-width

Detect unrecognized data source options

Aperta
#9 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Scala
Stelle
4
Fork
0
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Background
Options are passed to Spark data sources as key-value pairs. It is very easy to make a typo in an option name. If the data source has a default behaviour for this option (e.g. the option is not mandatory), the passed option will be unrecognized and therefore ignored.

It would be very useful if the data source would detect unrecognized options.

## Feature
1. Detect unrecognized options and log them as warnings.
2. Add `.option("pedantic", true)`, which, when enabled, will fail the Spark Application if there is at least one unrecognized option passed to the source.

## Additional context
As an inspiration for possible implementation, Cobrix uses a wrapper that takes incoming options. All configuration is read from that wrapper instead of the original configuration. The wrapper records which options are queried at least once. At any given point it allows getting the list of options that haven't been used.
An advantage of such implementation is that if there are dependencies between options, e.g. a particular option is only used if some other option is enabled, it is tracked automatically.

https://github.com/AbsaOSS/cobrix/blob/master/spark-cobol/src/main/scala/za/co/absa/cobrix/spark/cobol/parameters/Parameters.scala

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.