AbsaOSS / AbsaOSS/pramen

Add a mode for initial ingestion of event data

Abierto
#350 0 comentarios 0 reacciones 0 asignados Ver en GitHub
DE enhancement Pramen-Scala
Lenguaje dominante
Scala
Estrellas
31
Forks
4
Merge medio
1 d 10 min
PR fusionados (30 d)
4

Descripción

## Background
When an event table is ingested initially, the history can be quite long.
Executing each event date one by one can have a big overhead and execute many queries against the source database.

## Feature
Add a mode for initial ingestion of event data.

## Example
TBD

## Proposed Solution
What we can do is load full table in one go to a temporary directory (cache), and then run pipeline tasks for each date found in the database.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.