SparkSubmitOperator fails to poll driver status in Spark Standalone cluster mode

### Under which category would you file this issue?

Providers

### Apache Airflow version

3.3.1

### What happened and how to reproduce it?

**Issue Description**
Using the Airflow SparkSubmitOperator Airflow submits an application to the spark endpoint and awaits the status of the completed job. Whether the job finished successfully or failed in Spark, the Airflow client receives something like this:

```
[2026-08-20 11:58:01] DEBUG - polling status of spark driver with id driver-20260820115651-0003
[2026-08-20 11:58:01] DEBUG - Poll driver status cmd: ['spark-submit', '--master', 'spark://:7077', '--status', 'driver-20260820115651-0003']
[2026-08-20 11:58:02] DEBUG - spark driver status log: 26/08/20 11:58:02 WARN RestSubmissionClient: Unable to connect to server spark://:7077.

...

AirflowException: Failed to poll for the driver status 10 times: returncode = 1
```

**Reproducible Steps**
* Spark Standalone Cluster deployment
* Spark-submit configuration:
* deploy-mode = cluster
* spark.standalone.submit.waitAppCompletion = true
* Connection configuration:
* connection type: spark-submit
* host: spark://
* port: 7077

1. Start a Dag which contains a task created by SparkSubmitOperator
2. wait until job finished (configure loglevel to DEBUG for more logs)

exactly related to https://github.com/apache/airflow/issues/20949 also discussed in https://github.com/apache/airflow/discussions/21799

### What you think should happen instead?

**Expected Behaviour**

The Airflow Client, if set up with `spark.standalone.submit.waitAppCompletion` should poll and receive the correct status from spark.

Interestingly, if the Spark job succeeded at the first try (on the spark server) and the Airflow Client fails, if the Airflow Client retries, it creates a spark status request, which works, gets the corrects spark job status and sets the Airflow job to compelted successfully

**Analysis and Background**
(Taken from my own comment in #20949)

A Spark Standalone deployment exposes two relevant endpoints:

* 7077: binary endpoint used for submitting applications
* 6066: REST endpoint, which is also used for driver status requests

After submitting an application, SparkSubmitHook polls the driver status using _build_track_driver_status_command():
https://github.com/apache/airflow/blob/f8b8461e8191f88e72ad8c05b248e0385c21db99/providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py#L671

The implementation currently branches depending on whether the configured master URL uses port 6066.
If the port is 6066, Airflow performs a REST request using curl. Otherwise, it polls using `spark-submit --master --status `

The problem is that when the application is submitted through the normal Spark Standalone binary endpoint (7077), Airflow also uses that master URL for status polling. However, `spark-submit --status` internally perform a REST request for the driver's status, which requires the REST endpoint on port 6066.
see: [RestSubmissionClient.scala](https://github.com/apache/spark/blob/8acb55b6a8447c563c7a87095a737ed1363999cf/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala#L87)

At the same time, Airflow's Spark submission logic only uses the binary spark-submit mechanism:

https://github.com/apache/airflow/blob/f8b8461e8191f88e72ad8c05b248e0385c21db99/providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py#L650

Therefore, the configured Spark connection needs to point to the binary endpoint (7077) for submission, while status polling effectively needs the corresponding REST endpoint (6066).

To add to that, even if the polling url is somehow configurable in Airlfow to point to port 6066, the spark submit Client will create a hardcoded http request (see [RestSubmissionClient.scala - getBaseUrl](https://github.com/apache/spark/blob/8acb55b6a8447c563c7a87095a737ed1363999cf/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala#L420)). Especially in our use case, where we try to put the REST endpoint behind https, we wouldnt be able to access it via spark submit status polling.

**Related development in Airflow 3.3**
I also noticed that recent development in the Spark provider may already solve part of this problem at the operator level.
In particular, _StandaloneSparkSubmitBackend contains logic for checking the status of an existing Spark driver:

https://github.com/apache/airflow/blob/f8b8461e8191f88e72ad8c05b248e0385c21db99/providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_submit.py#L204

Starting with Airflow 3.3, this is used during task retries to determine whether a previously submitted Spark application actually succeeded.

From my understanding, this code already derives the appropriate REST status URL from the configured Spark master URL, even when the master URL points to the binary endpoint.

If that understanding is correct, perhaps the same logic could also be reused by SparkSubmitHook for normal driver status polling instead of determining the polling mechanism based on whether the configured port is 6066.

Our personal workaround is to override some of the spark hook's methods to inject the correct REST url. Other users probably take a similiar approach. I am sure though, there is a more elegant method to implement or refactor the hook to be able to call the REST Endpoint correctly.

**Contributing**
I'm not very familiar with the Airflow codebase or its contribution process, but if the maintainers agree with my findings, I would be happy to work on a patch with some guidance.

### Operating System

_No response_

### Deployment

Docker-Compose

### Apache Airflow Provider(s)

apache-spark

### Versions of Apache Airflow Providers

apache-airflow-providers-apache-spark==6.3.1

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Start in providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py at _build_track_driver_status_command() and the submission logic, then compare it with _StandaloneSparkSubmitBackend in operators/spark_submit.py. Confirm how a binary 7077 master maps to the REST status endpoint, including the cited Spark RestSubmissionClient behavior. Done means SparkSubmitOperator polls and reports the driver result correctly in standalone cluster mode.

Written by the indexing model from the issue text.