0xMiden / 0xMiden/node

Document handling idle gRPC client connections

Open
#2,366 3 comments 0 reactions 0 assignees View on GitHub
documentation rpc
Dominant language
Rust
Stars
104
Forks
138
Avg merge
1d 13h
Merged PRs (30d)
56

Description

gRPC uses `http2` connections to multiplex all client requests onto a single connection. It assumes that a client will connect, do a bunch of work, and then disconnect.

This is fine but leaves an attack vector where clients simple leave the connection idle, consuming a connection permanently without doing any work.

We currently "solve" this by auto-disconnecting every connection after 30 minutes. This is fine for general request/response connections since these shouldn't have long lived connections.

The problem with this is that we now have "external" connections for block sync streams. The 30minute disconnect has bitten us already on several occasions, though perhaps this a good thing to harden ourselves against these unexpected failure modes.

Context aside, I would like to investigate what we can do to truly prune _idle_ connections, and not just all connections after 30 minutes.

https://github.com/grpc/grpc-rust/issues/258 sort of sounds like this is an automatic thing? However it also sounds like it keeps connections alive if clients send `keep_alive` pings. Ideally we would only keep it if the client actively sends requests.

-- edit --

I have since discovered [MAX_CONNECTION_IDLE](https://grpc.io/docs/guides/keepalive/#keepalive-configuration-specification) which is exactly what we want, but is not supported by `tonic`.

afaict we can't do this without patching `tonic`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.