AbsaOSS / AbsaOSS/pramen

Add a mode for initial ingestion of event data

Open
#350 0 comments 0 reactions 0 assignees View on GitHub
DE enhancement Pramen-Scala
Dominant language
Scala
Stars
31
Forks
4
Avg merge
1d 10m
Merged PRs (30d)
4

Description

## Background
When an event table is ingested initially, the history can be quite long.
Executing each event date one by one can have a big overhead and execute many queries against the source database.

## Feature
Add a mode for initial ingestion of event data.

## Example
TBD

## Proposed Solution
What we can do is load full table in one go to a temporary directory (cache), and then run pipeline tasks for each date found in the database.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.