Reverse Tunnel Support for Envoy Gateway

*Summary*

We want to add native support for [Envoy Reverse Tunnels](https://www.envoyproxy.io/docs/envoy/latest/configuration/other_features/reverse_tunnel) to Envoy Gateway.

*Description*:
We recently introduced reverse tunnel support in Envoy Proxy v1.36, enabling bidirectional communication with services behind NAT, firewalls, or in private networks. In this pattern, an Envoy instance in a private network (initiator) establishes persistent TCP connections to an Envoy instance in an public network (responder). These connections are cached and reused to route traffic from the responder back to services behind the initiator. More details on the design and architecture are in [this Github issue](https://github.com/envoyproxy/envoy/issues/33320) and the [envoy docs](https://www.envoyproxy.io/docs/envoy/latest/configuration/other_features/reverse_tunnel).

Reverse tunnels are driven by custom socket interfaces , and require configuring multiple Envoy resources on both initiator and responder envoy:

**On the Initiator envoy:**
- Bootstrap extension to register the [reverse tunnel downstream socket interface](https://www.envoyproxy.io/docs/envoy/latest/configuration/other_features/reverse_tunnel#downstream-socket-interface)
- A [special listener](https://www.envoyproxy.io/docs/envoy/latest/configuration/other_features/reverse_tunnel#reverse-tunnel-listener) that initiates reverse connections to the responder

**On the Responder envoy**
- Bootstrap extension to register the [reverse tunnel upstream socket interface](https://www.envoyproxy.io/docs/envoy/latest/configuration/other_features/reverse_tunnel#upstream-socket-interface)
- A listener with a special network filter called "[reverse tunnel filter](https://www.envoyproxy.io/docs/envoy/latest/configuration/other_features/reverse_tunnel#upstream-socket-interface)" to accept reverse connections from the initiator
- [Clusters of type `REVERSE_CONNECTION`](https://www.envoyproxy.io/docs/envoy/latest/configuration/other_features/reverse_tunnel#upstream-socket-interface) to route data requests through cached reverse connections instead of forward connections.

As shown above, configuring reverse tunnels requires coordinating multiple Envoy resources across both initiator and responder envoys. Manually configuring these components using multiple low-level resources would be error-prone and not user-friendly.

To address this, we propose a new *ReverseTunnelConfig* CRD that allows users to configure all reverse tunnel metadata in a single place. With the ReverseTunnelConfig, please find below a rough example of how the reverse tunnels config would look like on the initiator and responder.

We have an internal PoC for this, and wanted to ask for design feedback by the envoy gateway community before working on the implementation. We had the following questions:

1. *New CRD for ReverseTunnelConfig*: We propose adding a new ReverseTunnelConfig CRD that allows configuring all reverse tunnel resources in one place. This would involve adding new types [here](https://github.com/basundhara-c/gateway/tree/main/api/v1alpha1). We wanted to ask for feedback/suggestions on this approach, and ask if the approach is acceptable?

2. For designating backends that should use reverse connection clusters, we see two possible approaches:

- Option 1: Define a Backend CRD with type ReverseTunnel; this is reflected in the config below and mirrors the pattern used for [dynamic forward clusters](https://github.com/basundhara-c/gateway/blob/d78c894aef8cd0bf1a623d7f4ef87bf0f5d23cc8/api/v1alpha1/backend_types.go#L242)

- Option 2: We can use a regular Kubernetes Service with a custom annotation like below. This will be doable with less code changes.

```
apiVersion: v1
kind: Service
metadata:
name: cluster-backend
namespace: default
annotations:
gateway.envoyproxy.io/backend-type: "reverse-tunnel"
gateway.envoyproxy.io/cluster-id: "initiator-envoy-cluster"
spec:
type: ClusterIP
clusterIP: None
ports:
- name: http
port: 80
```

We'd love to hear your feedback on what the preferred approach should be!

*INITIATOR ENVOY CONFIG*

```
# Initiator envoy config

---
# Gateway for the initiator
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: initiator-envoy-cluster
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ReverseTunnelConfig
metadata:
name: initiator-envoy
namespace: default
spec:
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: initiator-envoy-cluster

# Initiator metadata
initiator:
identity:
nodeId: initiator-envoy
clusterId: initiator-envoy
tenantId: nutanix

upstreams:
- name: responder-envoy
host: responder-envoy.envoy-gateway-system.svc.cluster.local
port: 9000
connections: 2

# Downstream services accessible through this reverse tunnel
# We can route everything to the gateway itself, where
# HTTPRoutes can be configured to route traffic to specific
# services.
services:
- name: envoy-default-cluster-1-34bd578c # service fronting the gateway
namespace: envoy-gateway-system
port: 80
path: /
```

*RESPONDER ENVOY CONFIG*

```
---
# The Gateway being configured
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: responder-gateway
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80

---
# ReverseTunnelConfig attaches to the Gateway
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ReverseTunnelConfig
metadata:
name: responder-config
namespace: default
spec:
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: responder-gateway # ← References the Gateway above

responder:
port: 9000

---
# Backend for reverse tunnel routing
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
name: reverse-tunnel-backend
namespace: default
spec:
type: ReverseTunnel # Custom backend type for reverse tunnels
reverseTunnel:
identifier:

---
# HTTPRoute also references the same Gateway
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: reverse-tunnel-route
namespace: default
spec:
parentRefs:
- name: responder-gateway

rules:
- matches:
- path:
type: PathPrefix
value: /reverse_tunnel
backendRefs:
- group: gateway.envoyproxy.io
kind: Backend
name: reverse-tunnel-backend
---
```

[optional *Relevant Links*:]
- [Envoy docs on reverse tunnels](https://www.envoyproxy.io/docs/envoy/latest/configuration/other_features/reverse_tunnel#)
- [Blog](https://www.nutanix.com/tech-center/blog/designing-secure-cross-cluster-communications-for-nutanix-part2-reverse-tunnel-initiation) on reverse tunnels architecture
- [Github issue](https://github.com/envoyproxy/envoy/issues/33320) for reverse tunnels