AbsaOSS / AbsaOSS/enceladus

Add performance summary to UI

Đang mở
#48 6 bình luận 0 reaction 1 người được giao Được @yruslan nhận Xem trên GitHub
feature help wanted Menas priority: low
Ngôn ngữ chính
Scala
Star
33
Fork
16
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

It would be helpful to add some performance metrics to Run screen, such as:

* input data size (in GB)
* output data size (in GB)
* Time it took to do standardization
* Time it took to do dynamic conformance
* Total processing time
* For each checkpoint it would be helpful to show the elapsed time as a difference between start and end processing time, if available (for source and raw it may not be available).
* Maybe also the number of records (they already shown as control measurements), could be part of performance summary as well.

An input data size and an output data size should be provided by backend. All other information can be derived from control checkpoints i guess.
There is a generic key/map pair in _INFO file. I guess we can use that to provide information about size of raw, standardized and published datasets. Also, possibly it should include information about standardization and conformance elapsed time.

The raw folder size, std folder size and conformed folder size can be saved in metadata fields of an _INFO file and Run object during std/dynconf jobs execution.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.