UDP routing remains active after deleting UDPRoute

Hi. I have a `Gateway` with exactly one listener. This listener is of the `protocol` UDP and a `UDPRoute` references it to route the UDP traffic to a coredns service (similiar to the [UDP Routing](https://gateway.envoyproxy.io/latest/tasks/traffic/udp-routing/) guide in the documentation). This works as expected, however, when I delete the `UDPRoute` again the routing remains active and I am still able to reach the coredns service using `dig`. Only the deletion of the `Gateway` itself stops the UDP routing.

On `Gateway`s with more than a single UDP listener this issue does not occur.

Steps to reproduce:

1. Deploy envoy gateway to a cluster
2. Create a `Gateway` with a UDP listener and no other listeners on that `Gateway`
3. Create a `UDPRoute` to route traffic via that `Gateway` to the coredns service (like in the [UDP Routing](https://gateway.envoyproxy.io/latest/tasks/traffic/udp-routing/) guide)
4. Use `dig` to confirm the proper functioning of the `UDPRoute`
5. Delete the `UDPRoute`
6. **Find UDP routing still active**

I use the envoy gateway **v1.2.4** (envoy **v1.32.2**) on a **v1.27** k8s cluster with the following config:

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
annotations:
name: eg
namespace: default
spec:
gatewayClassName: eg
listeners:
- allowedRoutes:
kinds:
- group: gateway.networking.k8s.io
kind: UDPRoute
namespaces:
from: Same
name: coredns
port: 5300
protocol: UDP
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
annotations:
name: coredns
namespace: default
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: eg
sectionName: coredns
rules:
- backendRefs:
- group: ""
kind: Service
name: coredns
port: 53
weight: 1
---
apiVersion: v1
kind: Service
metadata:
labels:
app: coredns
name: coredns
namespace: default
spec:
ports:
- name: udp-dns
port: 53
protocol: UDP
targetPort: 53
selector:
app: coredns
type: ClusterIP
```