AbsaOSS / AbsaOSS/enceladus

Add performance summary to UI

Abierto
#48 6 comentarios 0 reacciones 1 asignado Reclamado por @yruslan Ver en GitHub
feature help wanted Menas priority: low
Lenguaje dominante
Scala
Estrellas
33
Forks
16
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

It would be helpful to add some performance metrics to Run screen, such as:

* input data size (in GB)
* output data size (in GB)
* Time it took to do standardization
* Time it took to do dynamic conformance
* Total processing time
* For each checkpoint it would be helpful to show the elapsed time as a difference between start and end processing time, if available (for source and raw it may not be available).
* Maybe also the number of records (they already shown as control measurements), could be part of performance summary as well.

An input data size and an output data size should be provided by backend. All other information can be derived from control checkpoints i guess.
There is a generic key/map pair in _INFO file. I guess we can use that to provide information about size of raw, standardized and published datasets. Also, possibly it should include information about standardization and conformance elapsed time.

The raw folder size, std folder size and conformed folder size can be saved in metadata fields of an _INFO file and Run object during std/dynconf jobs execution.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.