api: Logical.writeRaw cancels the response body's context before the caller reads it (same defect as #18658, write/patch path)
- Dominant language
- Go
- Stars
- 36.3k
- Forks
- 4.8k
- PR merge metrics
- PR metrics pending
Description
**Describe the bug**
`(*api.Logical).WriteRawWithContext` (and `PatchRawWithContext`) can return a
`*Response` whose body is bound to an **already-cancelled** context. Reading the
body then fails with a spurious `context canceled`, even though the request
succeeded server-side.
Both public methods route through the unexported `writeRaw`
([api/logical.go](https://github.com/hashicorp/vault/blob/main/api/logical.go)):
```go
func (c *Logical) writeRaw(ctx context.Context, request *Request) (*Response, error) {
ctx, cancelFunc := c.c.withConfiguredTimeout(ctx)
defer cancelFunc() // fires before the caller reads resp.Body
resp, err := c.c.rawRequestWithContext(ctx, request)
return resp, err // body still unread, context now dead
}
```
The response body streams off the connection, so cancelling the context the
moment `writeRaw` returns can sever an in-flight body read. Whether a given call
survives depends on whether the body has already been buffered off the socket by
the time the caller reads it:
- **Small responses** are fully buffered before the cancel can matter, so the
read wins the race and the call succeeds. This is why the symptom is intermittent
for some requests.
- **Large responses** (still streaming at read time) lose the race
deterministically and fail with `context canceled`.
This is the exact defect that #18658 reported for the **read** path and that
PR #18708 fixed — by routing the raw read path through `RawRequestWithContext`
and leaving the context live until the caller is done. The raw **delete** path
(`DeleteRawWithContext`) and the raw **read** path (`readRawWithDataWithContext`)
both do this today. The raw **write/patch** path was never updated and still
cancels early.
The parsed `write` helper is *not* affected: it also installs the timeout cancel,
but it reads the body via `ParseSecret` **before** returning, so the cancel fires
only after the body is consumed. Only the raw write/patch path hands an unread
body back to the caller.
**To Reproduce**
The race only bites when the response body is still in flight at read time, so a
large write response makes it deterministic. A `transit` batch sign is a
convenient real write that returns a large body.
1. Start a dev server and create a signing key:
```sh
vault server -dev -dev-root-token-id=root &
export VAULT_ADDR=http://127.0.0.1:8200 VAULT_TOKEN=root
vault secrets enable transit
vault write -f transit/keys/sign-key type=ed25519
```
2. Run this client against it (default client config; its `Timeout: 60s` is what
installs the `withConfiguredTimeout` cancel the bug depends on):
```go
package main
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"sync"
"sync/atomic"
vaultapi "github.com/hashicorp/vault/api"
)
func main() {
cfg := vaultapi.DefaultConfig() // Timeout defaults to 60s
cfg.Address = "http://127.0.0.1:8200"
client, _ := vaultapi.NewClient(cfg)
client.SetToken("root")
// Large batch -> a few hundred KB of response body that cannot be fully
// buffered before ReadAll touches the connection.
in := base64.StdEncoding.EncodeToString([]byte("hello"))
batch := make([]map[string]string, 5000)
for i := range batch {
batch[i] = map[string]string{"input": in}
}
body, _ := json.Marshal(map[string]any{"batch_input": batch})
const n = 16
var ok, fail int64
var wg sync.WaitGroup
for i := 0; i < n; i++ {
wg.Add(1)
go func() {
defer wg.Done()
resp, err := client.Logical().WriteRawWithContext(
context.Background(), "transit/sign/sign-key", body)
if err != nil {
atomic.AddInt64(&fail, 1)
return
}
defer resp.Body.Close()
if _, err := io.ReadAll(resp.Body); err != nil {
atomic.AddInt64(&fail, 1) // fails with "context canceled"
return
}
atomic.AddInt64(&ok, 1)
}()
}
wg.Wait()
fmt.Printf("%d ok, %d failed (of %d)\n", ok, fail, n)
}
```
3. See every read fail:
```
0 ok, 16 failed (of 16) # every failure: body read error: context canceled
```
A single small sign (no `batch_input`) typically prints `1 ok, 0 failed` because
the tiny body is already buffered — which is exactly why the bug is intermittent
in practice and worsens under concurrency / larger payloads.
**Expected behavior**
`WriteRawWithContext` / `PatchRawWithContext` return a response whose body remains
readable, matching the raw read and delete paths. No spurious `context canceled`.
**Suggested fix:** mirror the read/delete fix from PR #18708 — have the raw write
path leave the context live for the caller (route the same way
`readRawWithDataWithContext` and `DeleteRawWithContext` do, via
`RawRequestWithContext`) rather than wrapping the request in a timeout-scoped
context that is cancelled before the body is read. This covers both
`WriteRawWithContext` and `PatchRawWithContext`, which share `writeRaw`. Locally
making that change flips the reproduction above from `0 ok, 16 failed` to
`16 ok, 0 failed` consistently.
**Environment:**
* Vault Server Version (retrieve with `vault status`): v1.18.3 (dev mode) — used only to exercise the client; the bug is purely client-side and reproduces against any server version.
* Vault CLI Version (retrieve with `vault version`): not relevant — bug is in the `github.com/hashicorp/vault/api` Go module, built at `main` (offending code verified present on `main`). Go 1.26.
* Server Operating System/Architecture: linux/amd64 (Ubuntu Jammy on WSL2 against local Vault, Jammy on VMware against real Vault)
Vault server configuration file(s):
```hcl
# N/A — reproduced with a stock `vault server -dev`; no custom server config required.
```
**Additional context**
`WriteRawWithContext` / `RawRequestWithContext` are marked deprecated in favor of
higher-level methods, but the parsed methods don't help callers that need the raw
response body, and the read-path fix (#18658 / #18708) already established that
the raw methods are expected to remain usable. The same expectation should hold
for the write/patch path.
References:
- #18658 — Logical.ReadRawWithDataWithContext cancels response body
- PR #18708 — the read-path fix this issue mirrors
Contributor guide
Research direction
Start in api/logical.go at writeRaw and compare it with readRawWithDataWithContext, DeleteRawWithContext, and RawRequestWithContext. Reproduce the issue with the large transit batch example, then verify that both WriteRawWithContext and PatchRawWithContext return bodies that can be fully read without a context-canceled error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100