tikv / tikv/pd

*: forwarding HTTP requests can fail with "net/http: cannot rewind body after connection loss"

Open
#10,590 1 comment 0 reactions 0 assignees View on GitHub
type/bugfix
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?

With audit or rate-limit middleware enabled, send HTTP requests to a PD
follower so that the follower forwards them to the PD leader via
`customReverseProxies` (e.g. `GET /pd/api/v1/cluster`).

When the forwarding `http.Client.Do` needs to retry on a new connection —
typically due to a stale keep-alive connection or HTTP/2 GOAWAY — the retry
fails.

### What did you expect to see?

The forwarded request is transparently retried on a new connection and
succeeds.

### What did you see instead?

The forward fails with:

```
[ERROR] [apiutil.go:465] ["request failed"] [error="[PD:http:ErrSendRequest]send HTTP request failed: Get \"https://.../pd/api/v1/cluster\": net/http: cannot rewind body after connection loss"]
```

Root cause: when audit/rate-limit middleware is enabled,
`requestutil.getBodyParam` replaces `r.Body` with
`io.NopCloser(bytes.NewBuffer(buf))` without setting `r.GetBody`. This turns
what was originally `http.NoBody` into a non-nil body reader with no rewind
function. On the forwarding path, `net/http` Transport probes the body
(`didRead=true`), and on connection loss cannot rewind because `GetBody` is
nil, so the retry path errors out with the message above.

### What version of PD are you using (`pd-server -V`)?

master (reproduces on current master and on the release branches that carry
the same audit middleware code path).

Contributor guide

Open the contributing guide

Research direction

Start by reading requestutil.getBodyParam and the forwarding path around apiutil.go:465, focusing on how audit or rate-limit middleware changes r.Body and how customReverseProxies invokes http.Client.Do. Reproduce with GET /pd/api/v1/cluster through a PD follower and a stale connection or HTTP/2 GOAWAY. Done means the forwarded request can retry on a new connection without the rewind error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend, networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.