EPS degrade when the number of workers is four times of CPU cores

In the benchmark comparing persistent queue (PQ) and memory queue, the EPS for 8 and 16 workers is similar. However, with 32 workers, the EPS for PQ significantly degrades compared to the memory queue. PQ does not receive enough events despite having available CPU resources. This suggests that thread contention, where worker threads block the write thread, may be the issue.

The logstash instance has 8 CPU cores and 8G memory.

| CPU | MEM | Worker | Batch Size | Queue | 1min EPS | 5min EPS | Worker Utilization | Worker Concurrency | Avg CPU% | Avg Heap | Avg Non Heap | Q Max Events | Q Max Size |
|-----|-----|--------|------------|-----------|----------|----------|--------------------|--------------------|----------|-------------|--------------|--------------|-------------|
| 8 | 8 | 8 | 1000 | persisted | 6172 | 6094 | 96.32 | 7.705 | 28 | 1733266033 | 181894718 | 168347 | 1073787947 |
| 8 | 8 | 16 | 1000 | persisted | 9790 | 9514 | 83.56 | 13.37 | 47 | 2356683605 | 183512033 | 797 | 132697837 |
| 8 | 8 | 32 | 1000 | persisted | 9253 | 9170 | 44.11 | 14.11 | 46 | 2463494258 | 184405256 | 30 | 147625218 |
| 8 | 8 | 8 | 1000 | memory | 5823 | 5585 | 99.99 | 8 | 21 | 2242772082 | 179471492 | 0 | 0 |
| 8 | 8 | 16 | 1000 | memory | 10230 | 9766 | 99.98 | 16 | 45 | 1976006473 | 178854446 | 0 | 0 |
| 8 | 8 | 32 | 1000 | memory | 16000 | 15550 | 99.99 | 32 | 71 | 2394150426 | 180885376 | 0 | 0 |

The issue names no files, tests, or entry points. Start by reproducing the benchmark with persisted and memory queues at 8, 16, and 32 workers, then investigate contention between worker threads and the persistent-queue write thread; done means identifying and addressing the cause of the EPS degradation.

Written by the indexing model from the issue text.