AbsaOSS / AbsaOSS/enceladus

Create runs for streaming jobs originating from Hyperdrive

Offen
#1,214 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Conformance feature priority: undecided under discussion
Vorherrschende Sprache
Scala
Sterne
33
Forks
16
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## Background
Now that Conformance can be run on streaming jobs from Hyperdrive we need to decide how we want to track these jobs as runs.

Currently, no runs are created for a streaming conformance job.

## Feature
We can create a separate entity for streaming jobs. My proposal is to call it `transmission`. It would contain only parameters that are relevant for streaming, such as
* Status (such as `running`, `stopped`, etc).
* A start timestamp of a streaming job
* A finish timestamp of a streaming job
* Number of records processed
* Average throughput
* Topic names
* Hyperdrive version and configuration options

Transmissions should have a separate tab in Menas where users can see what's happening.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.