The Celery Worker Pods terminated and lost task logs when scale down happened during HPA autoscaling

### Official Helm Chart version

1.15.0 (latest released)

### Apache Airflow version

2.7.1

### Kubernetes Version

1.27.2

### Helm Chart configuration

After official helm charts supports HPA, then here is my HPA settings.
```
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: "airflow-worker"
namespace: airflow
labels:
app: airflow
component: worker
release: airflow
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: StatefulSet
name: "airflow-worker"
minReplicas: 2
maxReplicas:16
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80

```

### Docker Image customizations

Not too much special packages, just added openjdk, mongo client and some other python related dependency.

### What happened

When I enabled HPA in the Airflow cluster, then during the scale down some long running tasks which might be over 15hrs got failed and lost the logs. More details.
- The long running task for example is calling python api to spin up a google dataproc cluster which will run the job over 15hrs. When the HPA started scale down the pods, then these tasks failed looks due to the pod terminated after the `terminationGracePeriodSeconds` was hitting. And then it caused the task failed, but actually the tasks still waiting the final response status from dataproc jobs.
- Also after the pod terminated, then the tasks' logs actually was not published to the GCS bucket. I set the remote log to GCS bucket already. But the logs actually are still in persistent disk side, I have to spin up the same pod then can upload the logs to GCS bucket. again.
- The `terminationGracePeriodSeconds` I set is 1hour right now, but due to the long running tasks so i am not sure how the HPA settings can support it.

### What you think should happen instead

It should be happened during HPA scaling down either the celery work pod need to graceful shutdown until all tasks are completed or Airflow should provide some config or something else to support kill the task process before final termination deadline and complete the cleanup and upload logs to remote settings like GCS bucket.

### How to reproduce

- Using official helm chart with HPA enabled with CPU/Memory metrics deploy airflow to Kubernetes.
- Setup remote logging
```
logging:
delete_local_logs: 'False'
remote_base_log_folder: "gs://xxx"
remote_logging: 'True'
```
- Run a long running task maybe just need 30mins or few hours, but you have to make sure setup `terminationGracePeriodSeconds` maybe to a small number.
- Try to trigger the HPA scale down the pods, then you should see the issue from the UI you should see the task failed without logs.

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

No source files or tests are named. Start by reproducing an HPA scale-down with the official Helm chart, a long-running Celery task, and GCS remote logging; then trace worker termination and log-upload behavior. Done means scale-down does not prematurely fail the task and task logs are uploaded to GCS before termination.

Written by the indexing model from the issue text.