Update latency increases with number of updates sent to workflow

Latency experiments (see https://github.com/dandavison/temporal-latency-experiments) show that update latency increases with the number of updates sent to the workflow. Example results:

The above measurements were collected in a us-west-2 EC2 instance. The "localhost" version was using temporal server start-dev (v1.25) and the "cloud" variant was using cloud namespace sdk-ci.a2dd6.tmprl.cloud (also in us-west-2). The latency measured is from sending an update to receiving the result. Go SDK. The update handler mutates local workflow state but does not block on anything. The experiments send n=2000 updates since this is the maximum allowed per workflow.

In contrast, neither signal latency nor query latency show the same trend:

Start by reviewing the latency experiments at https://github.com/dandavison/temporal-latency-experiments and reproducing the update, signal, and query comparisons described here. Trace the server paths for workflow updates and determine why update latency grows with the number of updates; done means the cause is identified and the behavior is covered by an appropriate regression test or benchmark.