Allow accepting new connections during a graceful shutdown
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 16.3k
- Forks
- 1.8k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 14
Description
Is your feature request related to a problem? Please describe.
We want a very specific behavior for graceful shutdowns in our hyper-based proxy. We set two deadlines, min (typically 5-10s), and max (could be much longer). We want the following sequence during shutdown:
- t=0s: send GOAWAY on all HTTP2 connections and start responding with
connection: closeon all http1 requests. Continue to accept new connections, and handle them the same (GOAWAY/connection: close) them. - After
mindeadline: stop accepting new connections. Continue existing connections, with the sameGOAWAY/connection: close. - After
maxdeadline: shutdown everything.
The reason for the time when we accept new connections, but tell clients to go away, is that there is some time for the load balancers in front of the proxy (this could be something like AWS NLB, Kubernetes kube-proxy, etc) to de-register the proxy instance and stop sending traffic to it. So immediately dropping new connections results in downtime. However, during this time its nice to start to encourage clients to goaway, since its likely they will select another proxy instance (either because the fronting LB already de-registered the old instance (the min deadline is intended to be the worst case, so usually its much faster), or because of random chance it selects a different one).
Today, the best we can do is wait until after min deadline and stop accepting new connections and start GOAWAY/connection: close at the same time.
Describe the solution you'd like
A new flag on hyper_util::server::conn::auto, and on hyper::server::conn::http1 to enable the "different graceful shutdown mode" (naming TBD!). This would change:
- https://github.com/hyperium/hyper/blob/f9f8f44058745d23fa52abf51b96b61ee7665642/src/proto/h1/dispatch.rs#L97 - skip the close() call
- https://github.com/hyperium/hyper/blob/f9f8f44058745d23fa52abf51b96b61ee7665642/src/proto/h1/conn.rs#L878 - skip the close() call
- https://github.com/hyperium/hyper-util/blob/66afc93debef02548c86e8454e6bc01cf4fca280/src/server/conn/auto/mod.rs#L341 - do no error here
- https://github.com/hyperium/hyper-util/blob/66afc93debef02548c86e8454e6bc01cf4fca280/src/server/conn/auto/mod.rs#L525-L531 -- if
cancelled, then callgraceful_shutdown()on the newly served connections (and do the same for UpgradeableConnection)
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
If these changes are acceptable I would be happy to contribute both parts of the change
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the referenced Rust locations in hyper's proto/h1/dispatch.rs and proto/h1/conn.rs, then hyper-util's server/conn/auto/mod.rs. Trace how cancellation and graceful shutdown affect new and upgradeable connections; done means the proposed flag supports accepting connections during the initial shutdown window while applying GOAWAY or connection: close, then stops accepting after the minimum deadline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- networking
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100