knative / knative/serving

queue-proxy can crash with concurrent map iteration and map write during HTTP/2 response finalization

Open
#16,656 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
6.1k
Forks
1.2k
Avg merge
2d 7h
Merged PRs (30d)
2

Description

## What version of Knative?

knative version:1.17.1

Observed in a downstream build based on Knative Serving. The queue-proxy logs report:

```json
"commit":"20c92a1-dirty"
```

The downstream commit resolves to `20c92a10d2ff6aa0162e0c74f9080e1fac7ec441`. The relevant timeout handler code matches current upstream `main` in the suspected critical path: `timeoutWriter.Header()` directly returns the underlying `http.ResponseWriter` header map without synchronization.

I also checked upstream `main` at `dbce551`; `pkg/http/handler/timeout.go` still contains:

```go
func (tw *timeoutWriter) Header() http.Header { return tw.w.Header() }
```

## Expected Behavior

`queue-proxy` should not crash due to concurrent HTTP/2 response header/trailer access. Request cancellation or timeout should fail only the affected request, not the queue-proxy process.

## Actual Behavior

`queue-proxy` crashed and restarted with `exitCode=2`. The fatal runtime error was:

```text
fatal error: concurrent map iteration and map write
```

The panic stack points to HTTP/2 response finalization iterating/promoting trailers while the response header map was concurrently modified:

```text
fatal error: concurrent map iteration and map write

goroutine 1123810148 [running]:
internal/runtime/maps.fatal({0x128031e?, 0xc000524530?})
runtime/panic.go:1058 +0x18
internal/runtime/maps.(*Iter).Next(0xc000948c20?)
internal/runtime/maps/table.go:683 +0x86
golang.org/x/net/http2.(*responseWriterState).promoteUndeclaredTrailers(0xc001d67780)
golang.org/x/net@v0.34.0/http2/server.go:2851 +0x65
golang.org/x/net/http2.(*responseWriterState).writeChunk(0xc001d67780, {0xc0008aa000, 0xf, 0x1000})
golang.org/x/net@v0.34.0/http2/server.go:2717 +0x77
golang.org/x/net/http2.chunkWriter.Write({0x10610e0?}, {0xc0008aa000?, 0xc0012831e0?, 0x1ce0680?})
golang.org/x/net@v0.34.0/http2/server.go:2670 +0x1d
bufio.(*Writer).Flush(0xc000b093c0)
bufio/bufio.go:643 +0x55
golang.org/x/net/http2.(*responseWriter).FlushError(0x142e0a0?)
golang.org/x/net@v0.34.0/http2/server.go:2935 +0x31
golang.org/x/net/http2.(*responseWriter).Flush(...)
golang.org/x/net@v0.34.0/http2/server.go:2925
golang.org/x/net/http2.(*responseWriter).handlerDone(0xc001425a50)
golang.org/x/net@v0.34.0/http2/server.go:3100 +0x25
golang.org/x/net/http2.(*serverConn).runHandler.func1()
golang.org/x/net@v0.34.0/http2/server.go:2473 +0x205
golang.org/x/net/http2.(*serverConn).runHandler(0x44cf12?, 0x0?, 0x0?, 0xc000146fb8?)
golang.org/x/net@v0.34.0/http2/server.go:2477 +0x109
created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 624989236
golang.org/x/net@v0.34.0/http2/server.go:2409 +0x21d
```

At the same timestamp, other goroutines were in the queue-proxy timeout / reverse-proxy handler path:

```text
knative.dev/pkg/network.ErrorHandler.func1
knative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/network/error_handler.go:33
net/http/httputil.(*ReverseProxy).ServeHTTP
net/http/httputil/reverseproxy.go:486
knative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP
knative.dev/serving/pkg/queue/request_metric.go:205
knative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3
knative.dev/serving/pkg/queue/handler.go:76
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2294
knative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4
knative.dev/serving/pkg/queue/forwarded_shim.go:54
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2294
knative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4
knative.dev/serving/pkg/http/handler/timeout.go:118
```

## Steps to Reproduce the Problem

I do not yet have a minimal standalone reproducer. The observed production trigger appears to involve:

1. queue-proxy serving HTTP/2 traffic.
2. Concurrent proxied requests.
3. Request cancellation or timeout while response handling is being finalized.
4. A response header/trailer map is modified while the HTTP/2 server iterates/promotes trailers during `handlerDone`.

## Suspected Cause

The crash looks like concurrent access to the underlying `http.Header` map owned by the HTTP/2 response writer.

`timeoutWriter` synchronizes `Flush`, `Write`, `WriteHeader`, and timeout error writes with `tw.mu`, but `Header()` returns the raw underlying header map without synchronization:

```go
func (tw *timeoutWriter) Header() http.Header { return tw.w.Header() }
```

Because callers can mutate the returned map outside `timeoutWriter`'s mutex, the HTTP/2 server may concurrently iterate the same map in `promoteUndeclaredTrailers`, causing:

```text
fatal error: concurrent map iteration and map write
```

This may be related to or another manifestation of #15850.

Contributor guide

Open the contributing guide

Research direction

Start with pkg/http/handler/timeout.go, especially timeoutWriter.Header and its callers, then trace queue/handler.go and the HTTP/2 finalization path shown in the stack. Reproduce or add focused concurrency coverage for cancellation or timeout during response finalization; done means queue-proxy no longer crashes and only the affected request fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.