AbsaOSS / AbsaOSS/spline

Postponed execution events linking

未關閉
#260 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
component: Persistence scaling / performance
主要語言
Scala
星號
667
分支
155
平均合併
3 天 6 小時
30 天內合併 PR
4

描述

When building a global lineage we need to traverse the graph through the boundary of many execution plans, from the affected executions to their predecessors and so forth. For the performance reasons (since an execution plan can be a super node), it's good to avoid looking up the affected executions by sorting and grouping at the boundary of every execution plan, but instead to have them linked directly.
Since the graph is built continuously and asynchronously, at the time of processing a new execution event we don't know if all its predecessors have arrived and processed already. So we cannot build the static links between them immediately. Instead we could do it in time windows, long enough to accommodate any reasonable time delays in the events arrival (once an hour / a day).

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。