AbsaOSS / AbsaOSS/pramen

Add the ability to have metastore partitioning column as a string, not a date

Abierto
#424 0 comentarios 0 reacciones 0 asignados Ver en GitHub
DS enhancement Pramen-Scala
Lenguaje dominante
Scala
Estrellas
31
Forks
4
Merge medio
1 d 10 min
PR fusionados (30 d)
4

Descripción

## Background
Currently, information date format and type are ignored when the metastore persistence format is 'delta'.

For example, here the date format will be ignored:
```hocon
pramen.metastore.tables = [
{
name = "table1"
description = "Table 1 description"
format = "delta"
path = ${base.path}/table1

information.date.column = "info_date"
information.date.format = "yyyyMM"
}
]
```

while here it will work:
```hocon
pramen.metastore.tables = [
{
name = "table1"
description = "Table 1 description"
format = "parquet"
path = ${base.path}/table1

information.date.column = "info_date"
information.date.format = "yyyyMM"
}
]
```

(the only difference is the storage format)

## Feature
Add the ability to have metastore partitioning column as a string, not a date.

## Example
--

## Proposed Solution
Pramen writes Parquet partitions directly, it constructs the path. For Delta we rely on the type system. So I guess we need to treat any date format that is not 'yyyy-MM-dd' as having `string` type.

Code to look: https://github.com/AbsaOSS/pramen/blob/f61438dc012f738de9dab5b2eb198b9914922642/pramen/core/src/main/scala/za/co/absa/pramen/core/metastore/peristence/MetastorePersistenceDelta.scala#L35-L35

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.