AbsaOSS / AbsaOSS/enceladus

Extending support for Streaming Conformance mapping rule to more than join conditions

Aperta
#1,532 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Conformance feature priority: undecided Streaming
Lingua principale
Scala
Stelle
33
Fork
16
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Background
The current implementation of streaming conformance supports up to 5 join conditions. Unfortunately, we didn't find a way to support an arbitrary amount of join conditions since each number of arguments needs to be a separate UDF.

## Feature
Increase the maximum number of join conditions that can be used in the mapping rule for streaming (broadcast mapping rule) to at least 10 (the exact number is to be decided).

## Proposed Solution [Optional]
In order to support more join conditions, a pair of UDF generators need to be implemented for each number of arguments. One UDF generator for the mapping, the other is for the error column.

UDFs are defined in `BroadcastUtils` as `getMappingLambdaParamX()` and `getErrorLambdaParamX()` (where X is 1 to 5).
After 6 to 10 are added they are invoked from `getMappingUdf()` which needs to be extended too.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start in BroadcastUtils by reading getMappingLambdaParamX() and getErrorLambdaParamX() for X=1..5, then inspect getMappingUdf() to trace how argument counts are selected. Confirm the agreed maximum before extending the supported range; done means the streaming broadcast mapping rule supports it through both the mapping and error UDF paths.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
scala, spark
Ambito
stream-processing
Tipo di issue
Funzionalità
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.