grpc / grpc/grpc-web

Text mode stops delivering data for large streaming calls

Open
#952 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
JavaScript
Stars
9.3k
Forks
802
Avg merge
1d 7h
Merged PRs (30d)
5

Description

I have an application that uses a server streaming call which sends back around 800mb of data to the client (206 onNext calls ~4mb each), given text/base64 encoding this equates to roughly 1.1GB on the wire.

When the stream starts, I can see the HTTP call in chrome developer tools, and the data returned by the endpoint grow in 4mb chunks over time, spending some time between chunks (presumably doing deserialization and handling my client code).

Roughly half way through, around the 300-400mb mark, it seems the generated client stops calling `on('data')`, and I get no further callbacks.

The rate at which chunks arrive/grow in chrome developer tools continues growing but at a much faster rate (as if it's not doing deserialziation anymore), and after all of the data arrives (some 10s later), I get an `on('end')` callback.

It seems something goes wrong in the client/it hits some sort of limit which breaks the mechanism used to call back to the caller.

There is no error/status/metadata callback to explain this.

For curiosity sake, I tried removing all of my code processing the data, and simply count the number of `on('data')` calls, and it seems I could get 96 of those calls (without doing anything with the data in the call) equating roughly 370mb of raw data.
I can still observe the rate of arrival accelerating, somewhat confirming that serialisation stops happening.

Lastly, on the text client, I tried monkey patching the format flag to use binary format (in the text client).

I know binary format does not support server side streaming, but it does seem to work, it's just the behaviour is different.
I get all of the data from the server side first, and only then it starts getting `on('data')` callbacks dispatched.
It seems in this mode, I do get all of the data and nothing is lost.

Any ideas what this could be? I can see that all of the data arrives to the browser from the network tab, so it's definately something with the client code.

It does not seem to be time related, but rather size related. I wonder if large data growth triggers some sort of GC which breaks things.

The boundary on which it stops sending data is also not deterministic, sometimes I get 95 RPCs done, sometimes 96, even though the data coming back is deterministic/fixed size.

Also, any ideas if there is any form of support for transport level compression when using envoy as the translation layer?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.