AbsaOSS / AbsaOSS/pramen

Allow sources to be configured as 'always have data'

オープン
#184 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
enhancement Pramen-Scala
主要言語
Scala
スター
31
フォーク
4
平均マージ
1日 10分
マージ済み PR(30日)
4

説明

## Background
Currently, ingestion jobs always checks the record count first, and only if the record count is non-zero, proceeds with the ingestion. This works well for event-based JDBC sources. But for file based sources it sometimes requires to parse data twice: the first time to get the record count, and the second time to load the data.

## Feature
Extend the interface of `Source` to allow instances where data is always assumed to be available, like snapshot sources.

When a source has 'always data available' the record count should be derived on write.

## Things to consider
- If the data does not get through the metastore, but passed directly from a source to a sink, possibly requires caching data, or saving in a temporary directory.

## Proposed Solution
```scala
trait Source extends ExternalChannel {
/**
* If true, getRecordCount() won't be used to determine if the data is available.
* This saves performance ono double data read when the source is file based and always points to a particular file,
* or it is a snapshot-based source
*/
def isDataAlwaysAvailable: Boolean = false
}
```

The default implementation is `false` so that the interface is source code compatible with existing sources.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。