AbsaOSS / AbsaOSS/enceladus

Extending support for Streaming Conformance mapping rule to more than join conditions

オープン
#1,532 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
Conformance feature priority: undecided Streaming
主要言語
Scala
スター
33
フォーク
16
PR マージ指標
30日以内にマージされた PR はありません

説明

## Background
The current implementation of streaming conformance supports up to 5 join conditions. Unfortunately, we didn't find a way to support an arbitrary amount of join conditions since each number of arguments needs to be a separate UDF.

## Feature
Increase the maximum number of join conditions that can be used in the mapping rule for streaming (broadcast mapping rule) to at least 10 (the exact number is to be decided).

## Proposed Solution [Optional]
In order to support more join conditions, a pair of UDF generators need to be implemented for each number of arguments. One UDF generator for the mapping, the other is for the error column.

UDFs are defined in `BroadcastUtils` as `getMappingLambdaParamX()` and `getErrorLambdaParamX()` (where X is 1 to 5).
After 6 to 10 are added they are invoked from `getMappingUdf()` which needs to be extended too.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start in BroadcastUtils by reading getMappingLambdaParamX() and getErrorLambdaParamX() for X=1..5, then inspect getMappingUdf() to trace how argument counts are selected. Confirm the agreed maximum before extending the supported range; done means the streaming broadcast mapping rule supports it through both the mapping and error UDF paths.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
scala, spark
領域
stream-processing
issue の種類
機能追加
難易度
3/5
見積もり時間
1〜2日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
45/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。