Ingestion tasks stuck on PENDING for a while

### Affected Version

We are using version 0.21.0

### Description

When running ingestion tasks we see that some of our tasks are stuck at "PENDING" state before starting even though there are no running tasks for the same interval or even the same datasource. Also, there are plenty of available slots in the middle-managers where we saw this error.

- Cluster size
We have ~30 middle-managers of different types and JVMs - We have different middle-managers for native ingestion/EMR ingestion/Kafka ingestion.
We run ~20k ingestion tasks a day.

- Configurations in use
We use a custom JavaScript affinity to determine where to navigate each task.

- The error message or stack traces encountered. Providing more context, such as nearby log messages or even entire logs, can be helpful.
We didn't encounter any error, the task is stuck at PENDING state for a while, no task log is available and we couldn't find any indicative message in the middle-manager logs.

- Any debugging that you have already done
* Searched middle-manager logs
* Searched task logs

Any help would be really appreciated!

Thanks,
Shachar

Start by reproducing the pending-task behavior in Apache Druid 0.21.0 with the reported middle-manager mix and custom JavaScript affinity. Review task scheduling and affinity behavior, using middle-manager and task logs to trace why eligible ingestion tasks do not start. Done means identifying the scheduling cause and adding a verified fix or a clearly documented reproduction and diagnosis.

Written by the indexing model from the issue text.