AbsaOSS / AbsaOSS/enceladus

Atum time measurements do not correspond to spark history

Offen
#910 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug priority: undecided
Vorherrschende Sprache
Scala
Sterne
33
Forks
16
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## Describe the bug
The time measurements logged in Run documents (e.g. Run start time, checkpoint start and finish ) do not match the time measurements in spark history. The total uptime (i.e. time experienced by users running the Spark app in Client mode) appear to be longer than presented in Runs. The difference is especially noticeable for small jobs (e.g. 1 min jobs appear several times shorter). The Atum measurements tend to not include: some of the jobs within the app (e.g. writing output), initial api calls to Menas and Spline, catalyst optimization time (?).

## To Reproduce
1. Run Enceladus job
2. Open event timeline in Spark history for the corresponding application
3. Open the corresponding Run document in mongodb
4. Compare times

## Expected behaviour
It is up for a discussion what should be included in the measured times:
- Total uptime experienced by the users (from call to exit of spark-submit). This might be important for building time-dependand pipelines
- Time of computation: the time between starting and finishing of all stages. This is important for performance monitoring and tuning.

## Screenshots
* Runs use UTC, spark history UI shown in CET
![Screen Shot 2019-09-19 at 10.40.53.png](https://images.zenhubusercontent.com/5bfe627bb82313279e7532e0/69df2a2a-9726-4d6c-b4ac-a57ee2a5a21f)

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.