AbsaOSS / AbsaOSS/pramen

Fail the full pipeline on certain types of exceptions

Open
#87 0 comments 0 reactions 0 assignees View on GitHub
enhancement Pramen-Scala
Dominant language
Scala
Stars
31
Forks
4
Avg merge
1d 10m
Merged PRs (30d)
4

Description

## Background
Sometimes the pipeline fails on an attempt to access a stopped Spark Session:
```
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
This stopped SparkContext was created at:

org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
za.co.absa.pramen.core.runner.PipelineSparkSessionBuilder$.buildSparkSession(PipelineSparkSessionBuilder.scala:66)
za.co.absa.pramen.core.runner.AppRunner$$anonfun$getSparkSession$1.apply(AppRunner.scala:111)
za.co.absa.pramen.core.runner.AppRunner$$anonfun$getSparkSession$1.apply(AppRunner.scala:111)
scala.util.Try$.apply(Try.scala:192)
za.co.absa.pramen.core.runner.AppRunner$.getSparkSession(AppRunner.scala:110)
za.co.absa.pramen.core.runner.AppRunner$.runPipeline(AppRunner.scala:54)
za.co.absa.pramen.runner.PipelineRunner$.main(PipelineRunner.scala:30)
za.co.absa.pramen.runner.PipelineRunner.main(PipelineRunner.scala)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
```

But the pipeline continues to run. This creates a lot of errors since every next transformation can't run.

## Feature
Make a list of fatal exceptions and fail the pipeline as soon as one of these is hit.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.