Perhaps longer latencies for Netherite for medium-sized payloads

We have switched to Netherite for our production workloads and we are quite pleased with the results in our testing.

However, it appears that for some size payload, the latency of calling orchestrators and activities is greater than with the same code using the default backend and we'd like to explore if there are ways to get this more in line.

Our main codebase is cored in Change feeds from Cosmos DB and the change feed immediately calls an orchestrator that then calls activity based on the state of the document. I have taken some basic A/B data for the time the change feed function takes and it appears the timing is longer for Netherite.

This is a simple test without being able to properly isolate all the factors, but wanted to know if any of you, especially @sebastianburckhardt might know of a reason. The below graph illustrates the longer latencies. The blue line represents the request duration for our main change feed for the standard backend, while the red is the duration when Netherite is used. The Netherite durations are spread over a wider range and overall at least twice the duration of the standard backend, it would seem. The data is normalized so that the scale is percentage of the total requests.

Thanks!

The issue names no source files or tests. Start by reproducing the A/B comparison around the Cosmos DB change feed, orchestrator, and activity path while isolating payload size and backend; done means identifying whether Netherite causes the reported latency difference and documenting a confirmed follow-up.