Fail the full pipeline on certain types of exceptions
- Dominant language
- Scala
- Stars
- 31
- Forks
- 4
- Avg merge
- 1d 10m
- Merged PRs (30d)
- 4
Description
## Background
Sometimes the pipeline fails on an attempt to access a stopped Spark Session:
```
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
This stopped SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
za.co.absa.pramen.core.runner.PipelineSparkSessionBuilder$.buildSparkSession(PipelineSparkSessionBuilder.scala:66)
za.co.absa.pramen.core.runner.AppRunner$$anonfun$getSparkSession$1.apply(AppRunner.scala:111)
za.co.absa.pramen.core.runner.AppRunner$$anonfun$getSparkSession$1.apply(AppRunner.scala:111)
scala.util.Try$.apply(Try.scala:192)
za.co.absa.pramen.core.runner.AppRunner$.getSparkSession(AppRunner.scala:110)
za.co.absa.pramen.core.runner.AppRunner$.runPipeline(AppRunner.scala:54)
za.co.absa.pramen.runner.PipelineRunner$.main(PipelineRunner.scala:30)
za.co.absa.pramen.runner.PipelineRunner.main(PipelineRunner.scala)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
```
But the pipeline continues to run. This creates a lot of errors since every next transformation can't run.
## Feature
Make a list of fatal exceptions and fail the pipeline as soon as one of these is hit.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.