AbsaOSS / AbsaOSS/pramen

Add timeout based on expected execution time of JDBC ingestion

未关闭
#732 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Scala
星标
31
派生
4
平均合并
1 天 10 分钟
30 天内合并 PR
4

描述

## Background
Sometime JDBC connections hang when running Pramen ingestion job.
It can result 1000 records from a DB to be loaded in hours.

## Feature
Add timeout based on expected execution time of JDBC ingestion.

## Example [Optional]
A simple example if applicable.

## Proposed Solution [Optional]
Define minimum execution time for JDBC insertions (even for 1 record), say 10 minutes by default.
Define minimum throughput, say 10 records per second by default.

When the record count is known in advance set the timeout based on above parameters.

Should the ingestion be restarted, or just stopped when the timer has expired - to be decided

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。