h2 library upgrade causes policy service to terminate connections from clients with a large number of policy watches
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 11.5k
- Forks
- 1.4k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 53
Description
### What is the issue?
The h2 upgrade from 0.4.15 to 0.4.17 in https://github.com/linkerd/linkerd2/pull/15592 and released in https://github.com/linkerd/linkerd2/releases/tag/edge-26.8.3 included this change from h2 library https://github.com/hyperium/h2/pull/935, which was released in h2 0.4.16.
The h2 change adds a "small DATA frame" budget which limits the number of small http2 DATA frames that a client can send to the server, in an attempt to protect the server from denial of service attacks. When the client overruns this budget by sending too many small http2 DATA frames, the server terminates the connection with a GOAWAY.
This change, however, is causing the policy container to terminate valid connections. linkerd client proxies can easily overrun the default small DATA frame budget in the h2 library by creating a few hundred policy watches. Each `Watch()` gRPC request results in a small DATA frame sent from the proxy to the policy service, eventually causing the policy service to terminate the connection.
I attempted to upgrade from edge-26.8.2 to edge-26.9.1 and immediately began running into this issue. Client proxies that have a large number of outbound destinations would send many `Watch()` requests to the policy service, until the policy service terminated the connection. A few minutes later, the client proxies would retry, only to have the connection terminated again once the small DATA frame budget was exhausted and the h2 library closed the connection.
This issue was documented in the h2 library here: https://github.com/hyperium/h2/issues/944. The h2 library has since introduced two changes:
1. https://github.com/hyperium/h2/pull/942 makes the small DATA frame budget configurable. There is more discussion about how the default value interacts poorly with gRPC streams in the comments here: https://github.com/hyperium/h2/pull/942#issuecomment-5356922419
2. https://github.com/hyperium/h2/pull/946 attempts to set a smarter default small DATA frame budget, but it's not clear that this default value would be much better for gRPC streams and the policy container case
I think the only solution is for linkerd to set a very large DATA frame budget, at least a budget large enough to handle the maximum # of supported policy watches over a single http2 connection.
### How can it be reproduced?
on edge-26.8.3 or later, have a client proxy create 100s to 1000s of `Watch()` requests to a single policy container over a single http2 connection. Observe the policy container terminate the connection.
### Logs, error output, etc
```
{"timestamp":"2026-09-14T13:35:46.685816Z","level":"WARN","fields":{"message":"Unexpected policy controller response; retrying with a backoff","grpc.status":"The service is currently unavailable","grpc.message":"connection error"},"target":"linkerd_app_outbound::policy::api","spans":[{"name":"outbound"},{"addr":"{REDACTED}","name":"proxy"}],"threadId":"ThreadId(1)"}
```
```
{"timestamp":"2026-09-14T13:35:46.685450Z","level":"WARN","fields":{"message":"Service failed","error":"endpoint {REDACTED}: channel closed"},"target":"linkerd_reconnect","spans":[{"name":"policy"},{"addr":"linkerd-policy.linkerd.svc.cluster.local:8090","name":"controller"},{"addr":"{REDACTED}","name":"endpoint"}],"threadId":"ThreadId(1)"}
```
### output of `linkerd check -o short`
```
linkerd-version
---------------
‼ cli is up-to-date
is running version 26.5.1 but the latest edge version is 26.9.2
see https://linkerd.io/2/checks/#l5d-version-cli for hints
control-plane-version
---------------------
‼ control plane is up-to-date
is running version 26.9.1 but the latest edge version is 26.9.2
see https://linkerd.io/2/checks/#l5d-version-control for hints
‼ control plane and cli versions match
control plane running edge-26.9.1 but cli running edge-26.5.1
see https://linkerd.io/2/checks/#l5d-version-control for hints
linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-b4cbcf886-48xpt (edge-26.9.1)
* linkerd-destination-b4cbcf886-ss8gg (edge-26.9.1)
* linkerd-destination-b4cbcf886-ts9pl (edge-26.9.1)
* linkerd-identity-747bd65bd9-299vf (edge-26.9.1)
* linkerd-identity-747bd65bd9-2njw9 (edge-26.9.1)
* linkerd-identity-747bd65bd9-nxhdh (edge-26.9.1)
* linkerd-proxy-injector-78ff9bb97b-294ch (edge-26.9.1)
* linkerd-proxy-injector-78ff9bb97b-f8mj6 (edge-26.9.1)
* linkerd-proxy-injector-78ff9bb97b-qwwc9 (edge-26.9.1)
* metrics-api-597f645d5-59x5z (edge-26.8.2)
* tap-59769bcb58-bpfdc (edge-26.8.2)
* tap-injector-599fb5ddb9-c4m2h (edge-26.8.2)
* web-fcdf9dd58-xfxxl (edge-26.8.2)
see https://linkerd.io/2/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
linkerd-destination-b4cbcf886-48xpt running edge-26.9.1 but cli running edge-26.5.1
see https://linkerd.io/2/checks/#l5d-cp-proxy-cli-version for hints
Status check results are √
```
### Environment
k8s version: 1.35.7-gke.1222000
Environment: GKE
linkerd version: edge-26.9.1
### Possible solution
I think that linkerd must use the configuration options added to h2 here https://github.com/hyperium/h2/pull/942 to create a small DATA frame budget large enough to support the maximum # of policy watches that are supported over a single http2 connection.
### Additional context
_No response_
### Would you like to work on fixing this bug?
maybe
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the policy service's gRPC Watch() connections and how the h2 0.4.17 small DATA frame budget is applied. Reproduce the issue with hundreds to thousands of watches on one HTTP/2 connection, then verify that the connection remains usable through the maximum supported watch count without GOAWAY termination.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, rust
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100