AbsaOSS / AbsaOSS/pramen

Add an option to not track late data for a table

Aperta
#282 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
enhancement Pramen-Scala
Lingua principale
Scala
Stelle
31
Fork
4
Merge medio
1g 10m
PR unite (30g)
4

Descrizione

## Background
When ingestion happens from an operation database in snapshot mode (no event/info dates), and the database was unavailable for a specific date, it might not make sense to load that data with previous information date since the new data with thew new information date is going to be loaded anyway, and it is going to have the same contents.

## Feature
Add an option to not track late data for a table.

## Example
```hocon
pramen.metastore {
tables = [
{
name = "table_name"
# ...

# If true, Pramen will try loading all previous partition/info dates
# If false, Pramen won't try filling gaps, will just load data to the current partition/information date
track.late.data = false
}
]
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.