Unexpected Pod while deleting gateways with merge gateways enabled

### *Description*:

There's some unexpected Pod during this process for deleting gateways with `mergeGateways` features on.

### *Repro steps*:

1. Apply the manifest from [quickstart](https://gateway.envoyproxy.io/latest/user/quickstart/)
2. Apply these config below, to create a GC named `mg` with MergeGateways feature on, and 3 GTWs with 3 HTTPRoutes attached to each GTW

kubectl apply -f mg.yaml

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: mg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: custom-proxy-config
namespace: envoy-gateway-system
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: envoy-gateway-system
spec:
mergeGateways: true
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: merged-eg-1
namespace: default
spec:
gatewayClassName: mg
listeners:
- allowedRoutes:
namespaces:
from: Same
name: http
port: 8080
protocol: HTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: merged-eg-2
namespace: default
spec:
gatewayClassName: mg
listeners:
- allowedRoutes:
namespaces:
from: Same
name: http
port: 8081
protocol: HTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: merged-eg-3
namespace: default
spec:
gatewayClassName: mg
listeners:
- allowedRoutes:
namespaces:
from: Same
name: http
port: 8082
protocol: HTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: hostname1-route
spec:
parentRefs:
- name: merged-eg-1
hostnames:
- "www.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /example
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: hostname2-route
spec:
parentRefs:
- name: merged-eg-2
hostnames:
- "www.example2.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /example2
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: hostname3-route
spec:
parentRefs:
- name: merged-eg-3
hostnames:
- "www.example3.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /example3

```

4. Everything is fine now

```bash
k get gc ✭ ✈
NAME CONTROLLER ACCEPTED AGE
envoy-gateway gateway.envoyproxy.io/gatewayclass-controller True 65m
mg gateway.envoyproxy.io/gatewayclass-controller True 2s

k get gtw ✭ ✈
NAME CLASS ADDRESS PROGRAMMED AGE
merged-eg-1 mg 172.18.255.200 True 4s
merged-eg-2 mg 172.18.255.200 True 4s
merged-eg-3 mg 172.18.255.200 True 4s

k get po -n envoy-gateway-system ✭ ✈
NAME READY STATUS RESTARTS AGE
envoy-gateway-6b8cdbfcdc-fbcb8 1/1 Running 0 66m # <- focus on this
envoy-mg-e9949d90-6d8596c878-ff5rb 1/1 Running 0 14s
```

5. `k delete -f mg.yaml`

6. There're some unexpected Pod showing up

```bash
k get po -n envoy-gateway-system ✭ ✈
NAME READY STATUS RESTARTS AGE
envoy-default-merged-eg-2-c7655c02-dd7c5d5d7-wp62r 0/1 Terminating 0 2s # <- unexpected
envoy-default-merged-eg-3-2a08f1cd-85dc9dbc6f-6wm7n 0/1 Terminating 0 2s # <- unexpected
envoy-gateway-6b8cdbfcdc-fbcb8 1/1 Running 0 66m
```

### *Environment*:

lattest