AbsaOSS / AbsaOSS/pramen

Allow specifying Spark resources per Python-Py Transformer

Aperta
#36 0 commenti 1 reazione 0 assegnatari Vedi su GitHub
enhancement help wanted Pramen-Py
Lingua principale
Scala
Stelle
31
Fork
4
Merge medio
1g 10m
PR unite (30g)
4

Descrizione

## Background

Currently, Python transformations run with Spark resource settings configured per environment.
This is good if the cluster has dynamic resource allocation enabled. But for clusters that don't have dynamic resource allocation enabled it is not possible to fine-tune resources for each transformer.

## Solution
Pramen-Py already supports specifying resources per transformer #30. Need to add the capability to Pramen.

## Example
```conf
{
name = "Python transformer"
type = "python_transformation"
python.class = "SomeTransformer"
schedule.type = "daily"

info.date.expr = "@runDate"

output.table = "transformed_python"

dependencies = [
{
tables = [ table1 ]
date.from = "@infoDate"
}
]

spark.config {
spark.executor.instances = 4
spark.executor.cores = 1
spark.executor.memory = "4g"
}
}
```
Generated YAML:
```yaml
run_transformers:
- info_date: 2022-02-18
output_table: transformed_python
name: SomeTransformer
spark_config:
spark.executor.cores: 1
spark.executor.instances: 4
spark.executor.memory: 4g
```

Docs on Spark config: https://spark.apache.org/docs/latest/configuration.html

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.