ksqldb not scaling beyond a certain number of requests even when hardware is not fully utilised

Hi Team,

We have 3 node r5.xlarge cluster and there we have created a source table and executing pull queries.
- Upto 3000 RPS we are able to serve within 30ms with CPU under 55% on each node
- However when we are executing 3500 RPS, latency is going 300ms - 600 ms and total throughput is ~3200 RPS and CPU is reaching till ~60% on each node.
In both cases, memory utilisation remain almost same which says memory is not playing any role here.

We checked network bandwidth / EBS throughput / PPS etc at instance level and none of these parameters are maxing out.

Attaching threadDump which we have taken in each run and also multiple threadDumps are taken in each run with fews seconds of gap.

File names having naming convention ksql-threadDumpX.hprof are dumps taken while load test of 3k requests per sec and latency was under acceptable limits.

File names having naming convention ksql-threadDumpX-3500.hprof are dumps taken while load test of 3.5k requests per sec and latency was going really high during this time and as discussed earlier there is not much increase in CPU/Memory during this time and so we have CPUs available.

[threadDump.zip](https://github.com/confluentinc/ksql/files/9176085/threadDump.zip)

Start with the attached threadDump.zip and compare the ksql-threadDumpX.hprof dumps from the 3,000-RPS runs with the ksql-threadDumpX-3500.hprof dumps from the high-latency runs. The work is complete when the scaling bottleneck is identified and a concrete corrective change is defined and validated against the reported throughput and latency.

Written by the indexing model from the issue text.