AbsaOSS / AbsaOSS/pramen

Add timeout based on expected execution time of JDBC ingestion

Aperta
#732 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Scala
Stelle
31
Fork
4
Merge medio
1g 10m
PR unite (30g)
4

Descrizione

## Background
Sometime JDBC connections hang when running Pramen ingestion job.
It can result 1000 records from a DB to be loaded in hours.

## Feature
Add timeout based on expected execution time of JDBC ingestion.

## Example [Optional]
A simple example if applicable.

## Proposed Solution [Optional]
Define minimum execution time for JDBC insertions (even for 1 record), say 10 minutes by default.
Define minimum throughput, say 10 records per second by default.

When the record count is known in advance set the timeout based on above parameters.

Should the ingestion be restarted, or just stopped when the timer has expired - to be decided

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.