K8S Spark Operator doesn't delete an older application with the same name anymore

### Apache Airflow version

2.6.2

### What happened

Changes to the SparkKubernetesOperator which introduced in https://github.com/apache/airflow/pull/21092 were reverted by the changes added in https://github.com/apache/airflow/pull/29977

Until 2.6.2 when a spark application was created with the same name of a previous application, the old application (pods etc') was automatically deleted and the new application started (with the warning of 'Deleted SparkApplication with the same name' in the logs).
After 2.6.2 this scenario is causing an error starting the new application (409 - Already exists) and the previous application needs to be manually deleted (there is no easy way to delete it programmatically from airflow dags).

Was this behavior change done on purpose?
I can suggest to add a parameter to control the desired behavior.

### What you think should happen instead

Changes to the SparkKubernetesOperator which introduced in https://github.com/apache/airflow/pull/21092 were reverted by the changes added in https://github.com/apache/airflow/pull/29977

### How to reproduce

Start a new spark application with the same name of the previous one, compare 2.6.2 vs 2.5.3

### Operating System

EKS

### Versions of Apache Airflow Providers

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Start with the SparkKubernetesOperator entry point and compare the behavior in Airflow 2.6.2 and 2.5.3, using the referenced pull requests as history. Reproduce the same-name application case and inspect the 409 response; done means the intended replacement behavior or configurable alternative is established and verified.

Written by the indexing model from the issue text.