Latency goes up sometimes

Hi ,
We have very strange problem as you can see most of the time latency is blew 500ms but suddenly it goes up to 60s ( option.ResponseTimeout = 60 ) , so obviously silo throw "I have been told I am dead, so this silo will stop! ..." error and k8 restart it.


All grains have [stateless(200)]
There is no blocking code , all calling uses await

There is many "Response did not arrive on time in 00:01:00 for message" but error does not contains information about "Total Enqueued=x; Total processed=x; ..."

Interesting point is when this error start total activation on grains is about 50 60 , so there is room for make new grains.

We also get these errors
"This silo has not received a probe request since ..."
"SiloHealthMonitor timer should have fired at .., which is 00:00:14.8624508 ago"

Sometimes it takes more than 2-3 seconds from the start of the request from the api to the execution of grain


Internal orleans trace

We use Orleans 3.5.0 and Kubernetes

I have no idea how figure out problem because it is on k8 pod will replace and I can not get memory dump.