Fail the full pipeline on certain types of exceptions
- Langage dominant
- Scala
- Étoiles
- 31
- Forks
- 4
- Merge moyen
- 1 j 10 min
- PR mergées (30 j)
- 4
Description
## Background
Sometimes the pipeline fails on an attempt to access a stopped Spark Session:
```
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
This stopped SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
za.co.absa.pramen.core.runner.PipelineSparkSessionBuilder$.buildSparkSession(PipelineSparkSessionBuilder.scala:66)
za.co.absa.pramen.core.runner.AppRunner$$anonfun$getSparkSession$1.apply(AppRunner.scala:111)
za.co.absa.pramen.core.runner.AppRunner$$anonfun$getSparkSession$1.apply(AppRunner.scala:111)
scala.util.Try$.apply(Try.scala:192)
za.co.absa.pramen.core.runner.AppRunner$.getSparkSession(AppRunner.scala:110)
za.co.absa.pramen.core.runner.AppRunner$.runPipeline(AppRunner.scala:54)
za.co.absa.pramen.runner.PipelineRunner$.main(PipelineRunner.scala:30)
za.co.absa.pramen.runner.PipelineRunner.main(PipelineRunner.scala)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
```
But the pipeline continues to run. This creates a lot of errors since every next transformation can't run.
## Feature
Make a list of fatal exceptions and fail the pipeline as soon as one of these is hit.
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.