AbsaOSS / AbsaOSS/enceladus

Extending support for Streaming Conformance mapping rule to more than join conditions

Aberta
#1,532 0 comentários 0 reações 0 responsáveis Ver no GitHub
Conformance feature priority: undecided Streaming
Linguagem predominante
Scala
Estrelas
33
Forks
16
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

## Background
The current implementation of streaming conformance supports up to 5 join conditions. Unfortunately, we didn't find a way to support an arbitrary amount of join conditions since each number of arguments needs to be a separate UDF.

## Feature
Increase the maximum number of join conditions that can be used in the mapping rule for streaming (broadcast mapping rule) to at least 10 (the exact number is to be decided).

## Proposed Solution [Optional]
In order to support more join conditions, a pair of UDF generators need to be implemented for each number of arguments. One UDF generator for the mapping, the other is for the error column.

UDFs are defined in `BroadcastUtils` as `getMappingLambdaParamX()` and `getErrorLambdaParamX()` (where X is 1 to 5).
After 6 to 10 are added they are invoked from `getMappingUdf()` which needs to be extended too.

Guia de contribuição

Abrir o guia de contribuição

Direção de pesquisa

Start in BroadcastUtils by reading getMappingLambdaParamX() and getErrorLambdaParamX() for X=1..5, then inspect getMappingUdf() to trace how argument counts are selected. Confirm the agreed maximum before extending the supported range; done means the streaming broadcast mapping rule supports it through both the mapping and error UDF paths.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
scala, spark
Domínio
stream-processing
Tipo de issue
Funcionalidade
Dificuldade
3/5
Tempo estimado
1-2 dias
Status de atividade
Estagnada
Clareza
Razoavelmente clara
Facilidade para iniciantes
45/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.