AbsaOSS / AbsaOSS/pramen

Add an option to not track late data for a table

未关闭
#282 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement Pramen-Scala
主要语言
Scala
星标
31
派生
4
平均合并
1 天 10 分钟
30 天内合并 PR
4

描述

## Background
When ingestion happens from an operation database in snapshot mode (no event/info dates), and the database was unavailable for a specific date, it might not make sense to load that data with previous information date since the new data with thew new information date is going to be loaded anyway, and it is going to have the same contents.

## Feature
Add an option to not track late data for a table.

## Example
```hocon
pramen.metastore {
tables = [
{
name = "table_name"
# ...

# If true, Pramen will try loading all previous partition/info dates
# If false, Pramen won't try filling gaps, will just load data to the current partition/information date
track.late.data = false
}
]
```

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。