envoy-gateway shutdown manager can cause proxy pods to be stuck draining

*Description*:

Using a Gateway with hundreds of envoy-proxy replicas under load, I notice a few envoy-proxy pods (e.g. 3 of 200) go unready over the course of days or weeks, accumulating over time.

```
NAME↑ PF READY STATUS RESTARTS CPU MEM %CPU/R %CPU/L %MEM/R %MEM/L IP NODE AGE
envoy-ingress-envoy-public-54051c0c-7f654966f9-8hdqm ● 1/2 Running 1 603 667 15 n/a 2086 n/a some-ip some-node 19d
```

Upon inspection, there are some common characteristics of these:

* The envoy-proxy container is actually "DRAINING"
* The shutdown-manager has a restart count of 1

```
Command:
envoy-gateway
Args:
envoy
shutdown-manager
State: Running
Started: Sun, 27 Apr 2025 08:07:07 -0700
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 25 Apr 2025 09:17:35 -0700
Finished: Sun, 27 Apr 2025 08:07:06 -0700
Ready: True
Restart Count: 1
Requests:
cpu: 10m
memory: 32Mi
Liveness: http-get http://:19002/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:19002/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Startup: http-get http://:19002/healthz delay=0s timeout=1s period=10s #success=1 #failure=30
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 3m10s (x885 over 72m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
```

```
Apr 28 14:31:19 node-name kubelet[2642]: I0428 14:31:19.128109 2642 prober.go:107] "Probe failed" probeType="Liveness" pod="ingress/envoy-ingress-envoy-public-54051c0c-66fd74494d-bh2pn" podUID="4b4ceb53-7974-43bb-b5f2-561de6c39670" containerName="shutdown-manager" probeResult="failure" output="Get \"http://10.130.194.14:19002/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
```

It seems like under normal circumstances, the `shutdown-manager` is designed so that when an envoy-proxy Pod is being terminated, the `shutdown-manager` container terminates and triggers the `envoy-proxy` container to start draining. Then eventually the Pod termination period ensures the pod is cleaned up.

However, if the `shutdown-manager` inadvertently shuts down on its own (e.g. possibly failing its container liveness check), that tells `envoy-proxy` to drain, but of course, nothing will actually cleanup the Pod. So even if its rare, over time you end up with a handful of envoy-proxy pods that are perpetually draining, and the shutdown-manager has a restart count of 1.

This seems like a bug? What's the intended graceful shutdown design? I know its second nature to put liveness checks on containers, but should the shutdown-manager actually have a liveness check - it seems like if it ever restarts, you enter this perpetual drain state.

Workaround for now is just to periodically scan across clusters and delete these 1/2 Ready proxy pods, they're replaced an work fine.

>What issue is being seen? Describe what should be happening instead of
the bug, for example: The expected value isn't returned, etc.

*Repro steps*:
> Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.

>**Note**: If there are privacy concerns, sanitize the data prior to
sharing.

*Environment*:
>Include the environment like gateway version, envoy version and so on.

Envoy Gateway v1.3.2
Envoy Proxy: envoy:distroless-v1.33.1

*Logs*:
>Include the access logs and the Envoy logs.

Looking at the "previous" `shutdown-manager` container logs. It's causing the envoy-proxy to start draining. Even though the Pod is no being replaced or anything.

```
2025-04-07T23:40:40.212Z INFO shutdown-manager envoy/shutdown_manager.go:75 starting shutdown manager 2025-04-27T15:05:50.380Z INFO shutdown-manager envoy/shutdown_manager.go:128 initiating drain with 10 second minimum drain period and 60 second timeout 2025-04-27T15:05:50.421Z INFO shutdown-manager envoy/shutdown_manager.go:219 total connections: 4267
```