Task is not able to be run - k8s Celery workers with stale resources

### Apache Airflow version

Other Airflow 2 version (please specify below)

### What happened

We are facing a quite interesting problem with our celery workers. **We run on AWS EKS (m-series nodes running Bottlerocket OS), Airflow v2.6.2**.

When the amount of running tasks is high (close to our max cap), we notice an increase in the celery workers memory usage followed by a stale period. At the same time, CPU utilization goes to zero.

During this time, the worker container seems to freeze (no logs are produced). Of the tasks that were running, almost all of them failed with the only log being:
```
[2023-08-31, 13:20:20 UTC] {taskinstance.py:1093} INFO - Dependencies not met for , dependency 'Task Instance State' FAILED: Task is in the 'failed' state.
[2023-08-31, 13:20:20 UTC] {local_task_job_runner.py:154} INFO - Task is not able to be run
```

It is important to note that this problem happens sometimes, but not every time. Which makes it even more confusing. Also, workers pods are not killed in the process, nor do they throw any exceptions.

Some of our configurations are:
```
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG = 32
AIRFLOW__CORE__PARALLELISM = 64
AIRFLOW__CELERY__WORKER_CONCURRENCY = 12
```

We are running at the moment with 5 celery workers, with no CPU or memory limits. The scheduler seems fine during those periods, producing regular logs. The only thing I can think of at the moment is that its probably due to a shortage of resources in the celery workers? However the worker concurrency limit should prevent a high usage of CPU/memory. Any ideas will be appreciated.

Some executor metrics on the same time period:

### What you think should happen instead

CPU and Memory resources should not go stale. Also, workers should keep logging as usual, instead of "freezing".

### How to reproduce

Deploy through Helm chart, using the `CeleryExecutor`. Run the maximum number of allowed tasks at the same time. However, the problem is not guaranteed to happen.

### Operating System

Bottlerocket OS

### Versions of Apache Airflow Providers

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

AWS EKS with k8s `v1.26.1` - Nodes (m-series) using Bottlerocket OS.

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Start with the CeleryExecutor configuration, the worker settings shown in the issue, and the `local_task_job_runner.py:154` log path. Reproduce the intermittent freeze under the Helm deployment on EKS at the stated concurrency, then use worker logs and executor metrics to identify the cause; done means workers continue logging and tasks no longer become stale or fail in this state.

Written by the indexing model from the issue text.