AbsaOSS / AbsaOSS/pramen

Add the ability to have metastore partitioning column as a string, not a date

Offen
#424 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
DS enhancement Pramen-Scala
Vorherrschende Sprache
Scala
Sterne
31
Forks
4
Ø Merge
1 T. 10 Min.
Gemergte PRs (30 T.)
4

Beschreibung

## Background
Currently, information date format and type are ignored when the metastore persistence format is 'delta'.

For example, here the date format will be ignored:
```hocon
pramen.metastore.tables = [
{
name = "table1"
description = "Table 1 description"
format = "delta"
path = ${base.path}/table1

information.date.column = "info_date"
information.date.format = "yyyyMM"
}
]
```

while here it will work:
```hocon
pramen.metastore.tables = [
{
name = "table1"
description = "Table 1 description"
format = "parquet"
path = ${base.path}/table1

information.date.column = "info_date"
information.date.format = "yyyyMM"
}
]
```

(the only difference is the storage format)

## Feature
Add the ability to have metastore partitioning column as a string, not a date.

## Example
--

## Proposed Solution
Pramen writes Parquet partitions directly, it constructs the path. For Delta we rely on the type system. So I guess we need to treat any date format that is not 'yyyy-MM-dd' as having `string` type.

Code to look: https://github.com/AbsaOSS/pramen/blob/f61438dc012f738de9dab5b2eb198b9914922642/pramen/core/src/main/scala/za/co/absa/pramen/core/metastore/peristence/MetastorePersistenceDelta.scala#L35-L35

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.