hyperium / hyperium/hyper

Allow accepting new connections during a graceful shutdown

Open
#3,961 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

B-rfc C-feature
Dominant language
Rust
Stars
16.3k
Forks
1.8k
Avg merge
1d 22h
Merged PRs (30d)
14

Description

Is your feature request related to a problem? Please describe.
We want a very specific behavior for graceful shutdowns in our hyper-based proxy. We set two deadlines, min (typically 5-10s), and max (could be much longer). We want the following sequence during shutdown:

  • t=0s: send GOAWAY on all HTTP2 connections and start responding with connection: close on all http1 requests. Continue to accept new connections, and handle them the same (GOAWAY/connection: close) them.
  • After min deadline: stop accepting new connections. Continue existing connections, with the same GOAWAY/connection: close.
  • After max deadline: shutdown everything.

The reason for the time when we accept new connections, but tell clients to go away, is that there is some time for the load balancers in front of the proxy (this could be something like AWS NLB, Kubernetes kube-proxy, etc) to de-register the proxy instance and stop sending traffic to it. So immediately dropping new connections results in downtime. However, during this time its nice to start to encourage clients to goaway, since its likely they will select another proxy instance (either because the fronting LB already de-registered the old instance (the min deadline is intended to be the worst case, so usually its much faster), or because of random chance it selects a different one).

Today, the best we can do is wait until after min deadline and stop accepting new connections and start GOAWAY/connection: close at the same time.

Describe the solution you'd like
A new flag on hyper_util::server::conn::auto, and on hyper::server::conn::http1 to enable the "different graceful shutdown mode" (naming TBD!). This would change:

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context

If these changes are acceptable I would be happy to contribute both parts of the change

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the referenced Rust locations in hyper's proto/h1/dispatch.rs and proto/h1/conn.rs, then hyper-util's server/conn/auto/mod.rs. Trace how cancellation and graceful shutdown affect new and upgradeable connections; done means the proposed flag supports accepting connections during the initial shutdown window while applying GOAWAY or connection: close, then stops accepting after the minimum deadline.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.