apache / apache/airflow

error on _resolve_should_track_driver_status with spark standalone (cluster mode)

Open
#46,334 5 comments 0 reactions 0 assignees View on GitHub
_eol_affected_version:2.10 area:providers kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow Provider(s)

apache-spark

### Versions of Apache Airflow Providers

apache-airflow-providers-apache-spark 5.0.0
pyspark 3.5.4

### Apache Airflow version

2.10.3

### Operating System

ubuntu server 22.04

### Deployment

Other

### Deployment details

NA

### What happened

Hi

SparkSubmitOperator fails when interacting with a spark standalone server in deploy-mode=cluster.

In such condition, the _resolve_should_track_driver_status function in /usr/local/lib/python3.12/site-packages/airflow/providers/apache/spark/hooks/spark_submit.py returns True...

I have exactly the same pb as the one discussed here :
https://github.com/apache/airflow/discussions/21799

I did find a case opened for this issue.

### What you think should happen instead

I have an error when _resolve_should_track_driver_status returns True.
If I modify the function to return False, testing the task passes but in such case the final spark job status is not confirmed by the spark driver.

### How to reproduce

```
cat sparkoperator_test.py
import airflow
from datetime import datetime, timedelta

from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator

default_args = {
'owner': 'xxxxxxxx',
'start_date': datetime(2025, 1, 31),
}

with airflow.DAG('sparkoperator_test',
default_args=default_args,
schedule_interval='@hourly',
description='test of sparkoperator in airflow by Omar and Regis',
) as dag:

spark_conf={
'spark.standalone.submit.waitAppCompletion':'true',
}

spark_compute_pi = SparkSubmitOperator(
task_id='spark-compute-pi',
conn_id='spark_qa1',
java_class='org.apache.spark.examples.SparkPi',
application='/opt/spark/examples/jars/spark-examples_2.12-3.5.3.jar',
name='compute_pi_from_airflow',
application_args=["30"],
total_executor_cores=4,
executor_memory='1g',
executor_cores=1,
verbose='true',
conf=spark_conf,
execution_timeout=timedelta(minutes=10),
retries=0,
)

spark_compute_pi

```

spark version : 3.5.3
1 single node (hosting the master + 1 worker node)

The error happens when testing the task with `airflow tasks test ...`

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with _resolve_should_track_driver_status in airflow/providers/apache/spark/hooks/spark_submit.py and reproduce the failure using the SparkSubmitOperator example in sparkoperator_test.py with Spark standalone cluster deploy mode. Compare tracking behavior when it returns True or False; done means the task handles the driver status correctly and confirms the final Spark job status.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.