AbsaOSS / AbsaOSS/spline

Postponed execution events linking

Đang mở
#260 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
component: Persistence scaling / performance
Ngôn ngữ chính
Scala
Star
667
Fork
155
Merge trung bình
3 ngày 6 giờ
Pull request đã merge (30 ngày)
4

Mô tả

When building a global lineage we need to traverse the graph through the boundary of many execution plans, from the affected executions to their predecessors and so forth. For the performance reasons (since an execution plan can be a super node), it's good to avoid looking up the affected executions by sorting and grouping at the boundary of every execution plan, but instead to have them linked directly.
Since the graph is built continuously and asynchronously, at the time of processing a new execution event we don't know if all its predecessors have arrived and processed already. So we cannot build the static links between them immediately. Instead we could do it in time windows, long enough to accommodate any reasonable time delays in the events arrival (once an hour / a day).

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.