Proxy memory leak over HTTP 1

Nobody has claimed this yet.

### What is the issue?

In one of our production services, we see a memory leak that grows slowly over time but ends up using up to the 500MB limit before it is finally killed after running out of memory.

The service topology looks like so:

`Istio Ingress Gateway + Linkerd Sidecar` → http/2 → `Varnish + Linkerd Sidecar` → http/1.1 → `Workload + Linkerd Sidecar`

Below is a screenshot from our metrics of the linkerd-proxy sidecar memory usage of the `Workload` service in MiB:


As you can see the proxy seems to be continuously allocating memory, which is unlike the other services running with the linkerd proxy where it usually sits around 20-30MB

We have managed to get a heap profile of the linkerd proxy via bytehound and the majority of the allocations are in `bytes::bytes_mut::BytesMut::reserve_inner` which seems to be related to http1 keep alive based on the call graph in the following screenshot. This seems similar to an issue raised previously in the tokio bytes repository https://github.com/tokio-rs/bytes/pull/560 and a similar issue was raised in hyper referencing this https://github.com/hyperium/hyper/issues/2924. From the `Cargo.lock` file for the linkerd proxy, it seems that version 1.2.1 for the bytes crate is already being used, which should have the fix in the above PR.


What is interesting is that it seems like the connections between the varnish server and the backend workload are stuck in an HTTP1 state rather than being automatically upgraded to HTTP2, even after having restarted both workloads. To demonstrate:

- First exec into the varnish pod and run a curl to the backend service over HTTP/1.1:

```jsx
# curl --http1.1 site-production:7000 -vv 1>/dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.0.34.97:7000...
* Connected to site-production (10.0.34.97) port 7000 (#0)
> GET / HTTP/1.1
> Host: site-production:7000
> User-Agent: curl/8.1.2
> Accept: */*
>
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0< HTTP/1.1 200 OK
< x-powered-by: Express
< cache-control: max-age=0
< pragma: no-cache
< expires: 0
< p3p: CP="NOI ADM DEV COM NAV OUR STP
< content-type: text/html; charset=utf-8
< content-length: 60984
< x-response-time: 1373.572ms
< date: Tue, 13 Jun 2023 16:26:38 GMT
<
{ [55563 bytes data]
100 60984 100 60984 0 0 43503 0 0:00:01 0:00:01 --:--:-- 43528
* Connection #0 to host site-production left intact
```

- The corresponding linkerd log line on the backend service:

```jsx
site-production-d868d9dd-ssvvz linkerd-proxy {"client.addr":"10.4.9.82:56416","client.id":"-","host":"site-production:7000","method":"GET","processing_ns":"175794","request_bytes":"","response_bytes":"60984","status":200,"timestamp":"2023-06-13T16:26:37.246457617Z","total_ns":"1376377061","trace_id":"","uri":"/","user_agent":"curl/8.1.2","version":"HTTP/1.1"}
```

- Next, via a different pod curl the backend service again:

```jsx
# curl --http1.1 site-production:7000 -vv 1>/dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.0.34.97:7000...
* Connected to site-production (10.0.34.97) port 7000 (#0)
> GET / HTTP/1.1
> Host: site-production:7000
> User-Agent: curl/8.1.2
> Accept: */*
>
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0< HTTP/1.1 200 OK
< x-powered-by: Express
< cache-control: max-age=0
< pragma: no-cache
< expires: 0
< p3p: CP="NOI ADM DEV COM NAV OUR STP
< content-type: text/html; charset=utf-8
< content-length: 60984
< x-response-time: 1283.291ms
< date: Tue, 13 Jun 2023 16:28:46 GMT
<
{ [7691 bytes data]
100 60984 100 60984 0 0 46104 0 0:00:01 0:00:01 --:--:-- 46095
* Connection #0 to host site-production left intact
```

- The corresponding linkerd access log for the above curl:

```jsx
site-production-d868d9dd-4hpxk linkerd-proxy {"client.addr":"10.4.25.242:36928","client.id":"default.default.serviceaccount.identity.linkerd-system.cluster.local","host":"site-production:7000","method":"GET","processing_ns":"214589","request_bytes":"","response_bytes":"60984","status":200,"timestamp":"2023-06-13T16:24:07.289773447Z","total_ns":"1391694628","trace_id":"","uri":"http://site-production:7000/","user_agent":"curl/8.1.2","version":"HTTP/2.0"}
```

The difference between the two requests is that the second seems to have been upgraded correctly to H2 as seen via the `version` entry in the log. It also looks like the client id has been picked up correctly from the source pod in the second request but not in the first. All pods in the above test were automatically injected with the linkerd proxy by the control plane and the backend service only accepts HTTP/1.1 requests.

It seems like the issue with the memory might not happen if the proxy-to-proxy connection could be successfully upgraded to http2?

Any thoughts or recommendations for debugging are welcome, thanks in advance.

### How can it be reproduced?

Unfortunately, I have not been able to reproduce the issue in any test environments, it only seems to be happening on this specific request route, through these specific services.

### Logs, error output, etc

See the issue description

### output of `linkerd check -o short`

```
Linkerd core checks
===================

linkerd-version
---------------
‼ cli is up-to-date
is running version 2.12.4 but the latest stable version is 2.13.4
see https://linkerd.io/2.12/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
is running version 2.12.4 but the latest stable version is 2.13.4
see https://linkerd.io/2.12/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-db7888cfd-bzgxf (stable-2.12.4)
* linkerd-destination-db7888cfd-rbrnh (stable-2.12.4)
* linkerd-destination-db7888cfd-zkbpr (stable-2.12.4)
* linkerd-identity-d788f947-7p6l9 (stable-2.12.4)
* linkerd-identity-d788f947-xthbw (stable-2.12.4)
* linkerd-identity-d788f947-xvs2s (stable-2.12.4)
* linkerd-proxy-injector-7cdf67b86d-9fs5c (stable-2.12.4)
* linkerd-proxy-injector-7cdf67b86d-h76jf (stable-2.12.4)
* linkerd-proxy-injector-7cdf67b86d-z56w5 (stable-2.12.4)
see https://linkerd.io/2.12/checks/#l5d-cp-proxy-version for hints

linkerd-ha-checks
-----------------
‼ pod injection disabled on kube-system
kube-system namespace needs to have the label config.linkerd.io/admission-webhooks: disabled if injector webhook failure policy is Fail
see https://linkerd.io/2.12/checks/#l5d-injection-disabled for hints

Status check results are √
```

### Environment

- K8s version: 1.25.8
- environment: GKE
- linkerd version: 2.12.4

### Possible solution

_No response_

### Additional context

_No response_

### Would you like to work on fixing this bug?

None

Start with the HTTP/1.1 curl commands, Linkerd access logs, and the proxy dependency versions recorded in Cargo.lock. Investigate the reported Varnish-to-workload route and compare its HTTP/1.1 and HTTP/2 behavior; reproduction is currently unavailable outside the described production path. Done would require a reproducible case and a confirmed explanation or fix for the memory growth.

Written by the indexing model from the issue text.