AbsaOSS / AbsaOSS/pramen

Fail the full pipeline on certain types of exceptions

Abierto
#87 0 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement Pramen-Scala
Lenguaje dominante
Scala
Estrellas
31
Forks
4
Merge medio
1 d 10 min
PR fusionados (30 d)
4

Descripción

## Background
Sometimes the pipeline fails on an attempt to access a stopped Spark Session:
```
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
This stopped SparkContext was created at:

org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
za.co.absa.pramen.core.runner.PipelineSparkSessionBuilder$.buildSparkSession(PipelineSparkSessionBuilder.scala:66)
za.co.absa.pramen.core.runner.AppRunner$$anonfun$getSparkSession$1.apply(AppRunner.scala:111)
za.co.absa.pramen.core.runner.AppRunner$$anonfun$getSparkSession$1.apply(AppRunner.scala:111)
scala.util.Try$.apply(Try.scala:192)
za.co.absa.pramen.core.runner.AppRunner$.getSparkSession(AppRunner.scala:110)
za.co.absa.pramen.core.runner.AppRunner$.runPipeline(AppRunner.scala:54)
za.co.absa.pramen.runner.PipelineRunner$.main(PipelineRunner.scala:30)
za.co.absa.pramen.runner.PipelineRunner.main(PipelineRunner.scala)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
```

But the pipeline continues to run. This creates a lot of errors since every next transformation can't run.

## Feature
Make a list of fatal exceptions and fail the pipeline as soon as one of these is hit.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.