Livy could intermittently returns batch job status as SUCCEED even Spark on Kubernetes actually fails

I run a Livy server on Kubernetes to submit Spark batch jobs via Airflow using the Livy REST API. However, even when Spark jobs fail due to driver or executor issues, Livy incorrectly shows the status as "SUCCEED" and returns a successful response. This leads Airflow to mark failed jobs as successful mistakenly.

One related issue I found was this - https://issues.apache.org/jira/browse/LIVY-896 which got fixed in the next release 0.8.0
We upgraded our Livy cluster from 0.7.0 to 0.8.0 but it didn't fix the issue in the Kubernetes ecosystem.

While reviewing the Livy code, we found that it returns an exit code of 0 (success) even when the driver pod fails in cluster mode. Ideally, the main pod should transition to an 'Error' state if the application fails.

Start with the Livy REST API path for Spark batch submissions in Kubernetes cluster mode and reproduce a driver or executor failure. Trace how the main pod failure becomes the returned exit code and status, then verify that Airflow receives a failed response rather than SUCCEED.

Written by the indexing model from the issue text.