Incremental latency on server-side stream

Hello!
Thanks for the great library!

We are streaming images captured by a camera to the browser (in our usecase we prefer streaming captured images rather than video) using grpc. The setup is the following:
Python grpc client -> server -> grpc-web server stream -> browser, where the python client also captures images and sends them as a `bytes` field.

We are capturing images at ~1 fps and at best seeing about 500 ms latency from when the image is captured to when its shown in the browser. We have also concluded that the time from when the server recieves and publishes images is insignificant.

However, after about 20-30 seconds of recieving the latency at the browser starts increasing rapidly, up to more than a minute. A current workaround is to reopen the stream at the browser when latency exceeds a certain treshold, but this is a bit brittle.

Any advice what might be the issue?