AbsaOSS / AbsaOSS/enceladus

Add performance summary to UI

オープン
#48 コメント 6 件 リアクション 0 件 担当者 1 名 @yruslan が担当を希望しています GitHub で見る
feature help wanted Menas priority: low
主要言語
Scala
スター
33
フォーク
16
PR マージ指標
30日以内にマージされた PR はありません

説明

It would be helpful to add some performance metrics to Run screen, such as:

* input data size (in GB)
* output data size (in GB)
* Time it took to do standardization
* Time it took to do dynamic conformance
* Total processing time
* For each checkpoint it would be helpful to show the elapsed time as a difference between start and end processing time, if available (for source and raw it may not be available).
* Maybe also the number of records (they already shown as control measurements), could be part of performance summary as well.

An input data size and an output data size should be provided by backend. All other information can be derived from control checkpoints i guess.
There is a generic key/map pair in _INFO file. I guess we can use that to provide information about size of raw, standardized and published datasets. Also, possibly it should include information about standardization and conformance elapsed time.

The raw folder size, std folder size and conformed folder size can be saved in metadata fields of an _INFO file and Run object during std/dynconf jobs execution.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。