AbsaOSS / AbsaOSS/pramen

Add a mode for initial ingestion of event data

未关闭
#350 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
DE enhancement Pramen-Scala
主要语言
Scala
星标
31
派生
4
平均合并
1 天 10 分钟
30 天内合并 PR
4

描述

## Background
When an event table is ingested initially, the history can be quite long.
Executing each event date one by one can have a big overhead and execute many queries against the source database.

## Feature
Add a mode for initial ingestion of event data.

## Example
TBD

## Proposed Solution
What we can do is load full table in one go to a temporary directory (cache), and then run pipeline tasks for each date found in the database.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。