AbsaOSS / AbsaOSS/spline

Postponed execution events linking

Aberta
#260 0 comentários 0 reações 0 responsáveis Ver no GitHub
component: Persistence scaling / performance
Linguagem predominante
Scala
Estrelas
667
Forks
155
Merge médio
3d 6h
PRs com merge (30d)
4

Descrição

When building a global lineage we need to traverse the graph through the boundary of many execution plans, from the affected executions to their predecessors and so forth. For the performance reasons (since an execution plan can be a super node), it's good to avoid looking up the affected executions by sorting and grouping at the boundary of every execution plan, but instead to have them linked directly.
Since the graph is built continuously and asynchronously, at the time of processing a new execution event we don't know if all its predecessors have arrived and processed already. So we cannot build the static links between them immediately. Instead we could do it in time windows, long enough to accommodate any reasonable time delays in the events arrival (once an hour / a day).

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.