etcd-io / etcd-io/etcd

embed: Close can return with active peer HTTP connections

Open
#22,389 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
2d 21h
Merged PRs (30d)
43

Description

### Bug report criteria

- [x] This bug report is not security related.
- [x] This is not a support request or question.
- [x] I have read the bug reporting guidelines.
- [x] I checked existing issues and did not find a duplicate.

### What happened?

`embed.Etcd.Close` can return while an accepted peer HTTP connection is still open.

Peer servers are given one second to shut down. When an active connection does not finish before that deadline, `http.Server.Shutdown` returns a context error and leaves the active connection open. The error is currently ignored, then the cmux listener is closed. Because cmux no longer owns connections already accepted by the HTTP server, such a connection remains open until its I/O deadline expires.

This can be reproduced with a peer request that has entered the HTTP handler but has not completed its request body. After `Etcd.Close` returns, reading from the client side of that connection still blocks until its deadline.

### What did you expect to happen?

`embed.Etcd.Close` should first allow peer connections to finish gracefully. If the one-second shutdown deadline expires, it should close the remaining active peer HTTP connections before returning. This matches the documented shutdown behavior: "After timeout, enforce remaining requests be closed immediately."

### How can we reproduce it (as minimally and precisely as possible)?

1. Start a single embedded etcd server.
2. Connect directly to its peer listener.
3. Send a valid `POST /raft` request with `Content-Length: 1` and `Expect: 100-continue`.
4. Wait for `HTTP/1.1 100 Continue`, but do not send the request body. This confirms that the connection is active in the peer HTTP handler.
5. Call `embed.Etcd.Close`.
6. Set a short read deadline on the client connection and read from it.

The read times out, showing that the peer connection remains open after `Etcd.Close` returns. Calling `http.Server.Close` when `Shutdown` returns an error closes the connection immediately.

### Anything else we need to know?

TCP peer connections use `timeoutConn`, which applies a five-second deadline to each read. Unix listeners bypass that wrapper; for the incomplete-body request in this reproduction, the peer HTTP server's five-minute `ReadTimeout` applies. Both are separate from the one-second graceful peer shutdown timeout, so neither ensures that the connection is closed when `Etcd.Close` returns.

This report concerns closing accepted peer HTTP connections. Closing their sockets does not itself wait for all request-handler goroutines to return.

### Etcd version

Reproduced at `main` commit `d848b43ae`. The same shutdown sequence is present in v3.7.1.

### Etcd configuration

Single-member embedded server using the default configuration, with local Unix client and peer listeners.

### Etcd debug information

Not applicable to the isolated embedded-server reproduction.

### Relevant log output

No error is logged because the error returned by `http.Server.Shutdown` is ignored.

Contributor guide

Open the contributing guide

Research direction

Start at embed.Etcd.Close and trace the peer HTTP shutdown sequence through http.Server.Shutdown and the cmux listener. Reproduce the incomplete POST /raft request described in the issue, then verify that connections still active after the one-second deadline are closed before Close returns. Confirm behavior for both TCP timeoutConn connections and Unix listeners without changing handler-goroutine waiting semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.