ConduitIO / ConduitIO/conduit

Ensure back-pressure in gRPC streams

Open
#211 1 comment 0 reactions 0 assignees View on GitHub
feature roadmap
Dominant language
Go
Stars
610
Forks
63
Avg merge
12h 28m
Merged PRs (30d)
57

Description

### Feature description

If we have a source plugin producing records faster than a destination plugin can process them, we need to ensure that we apply back-pressure and throttle down the speed of records being produced.

Using plugins in the built-in mode is not a problem, because we are using channels to send/receive records to/from the plugin. We control the size of the buffer in those channels (currently unbuffered) and back-pressure will automatically be applied.

It's more complicated in standalone mode. Connector plugins are communicating with Conduit via gRPC streams when running in standalone mode. Conduit does not wait for a record to be successfully processed by the destination plugin before sending the next record into the stream, instead, it allows the plugin to asynchronously process the records and send back an acknowledgment once it is done. gRPC by default dynamically changes the buffer size based on [bandwidth estimation](https://grpc.io/blog/grpc-go-perf-improvements/#bdp-estimation-and-dynamic-flow-control-window). Since the plugin is running on the same machine as Conduit latency will be near 0 and bandwidth will be very high, so we can assume that the buffer will grow to the maximum, which is 4MB. Because we have 2 plugins on a pipeline (source and destination) a pipeline can thus buffer 8MB of records before it starts applying back-pressure, which sounds quite a lot (e.g. in case a record is 1kB in size that equals 8000 records).

A solution to this would be to set the [initial window size](https://pkg.go.dev/google.golang.org/grpc#WithInitialWindowSize) of a gRPC stream to a fixed size. The question is what size would make sense? We should investigate this further and possibly set a fixed window size once we have benchmarks set up (#198) and can measure how that impacts the performance of Conduit.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.