Migrating Route off Service.ExternalName

How does Knative use Service.ExternalName

We currently create an ExternalName service for each Knative Route. This results in a CNAME DNS record making the Route accessible within the cluster. This path supports all the desired traffic splitting features etc.

The ExternalName value is typically sourced from the KIngress.Status.(Public|Private)LoadBalancer property. Current ingress implementations set a DomainInternal value for the LoadBalancer status which typically point to an internal K8s service pointing at the ingress's L7 Proxy. The other status that can be propagated back are: IP, Domain, and MeshOnly (bool)

The current DomainMapping implementation will create a KIngress object pointing to the Status.Address.URL property of the referenced object. For Knative Service/Route this ends up being the K8s ExternalName Service.

Problem

There's an advisory (https://github.com/kubernetes/kubernetes/issues/103675) suggesting ingress implementations should not forward traffic to a Service of type ExternalName and/or allow disabling this functionality.

This is exactly what Contour has done because of https://github.com/projectcontour/contour/security/advisories/GHSA-5ph6-qq5x-7jwc and I'm assuming long term more will follow suit providing options to disable.

Thus long term we should avoid the use of ExternalName services in order for DomainMapping to function correctly.

Potential Approaches

1) Changing DomainMapping implementation

1.1) Resolving ExternalName to the target Namespace

PoC PR: https://github.com/knative/serving/pull/11747

The approach here is to resolve the Status.Address.URL and create the KIngress object in the target namespace. This would require moving and keeping related resources in sync across namespaces. This is because certain ingress implementation don't allow you to reach across namespaces to access resource - ie. contour & k8s secrets.

1.2) Don't use KIngress

Since DomainMapping is essentially a Host/URI rewrite this can be done with a filter approach instead of using a KIngress.
The filter would rewrite the host & path etc prior to forwarding requests upstream.

We don't have abstractions for this type of functionality and whether it's supported by different ingresses. Given that maybe this is a feature request to bring to the upstream Gateway API folks.

2) Changing Route Service to not use ExternalName

2.1) Use Endpoints

If an Ingress implementation disables ExternalName forwarding they could populate the KIngress Status with an IP and the Route reconciler can opt to use a Headless Service with our own managed Endpoint. Endpoints don't support CNAME addresses so we'll still require Service ExternalName if the IP is not set.

Some observations/considerations:
Headless means switching the type to ClusterIP and setting the clusterIP property to None.
These properties are immutable (context: https://groups.google.com/g/kubernetes-sig-network/c/8opCQZhxpxM/m/CynHrF_cAQAJ). Also switch the type to ClusterIP => ExternalName seems broken see the issue: https://github.com/kubernetes/kubernetes/issues/104329

So if a an ingress implementation toggles between the IP it means we would have to delete the service/endpoint and recreate it when that happens. That's probably ok.

I think I prefer a single IP vs N IPs - this is to avoid cascading herds - ie. contour uses a daemon set so a large cluster would result in many IPs being propagated & when deployments are rolled vs. a stable service IP.

2.2) Use EndpointSlices

EndpointSlices support A and CNAME records- so we could create a headless service and a single EndpointSlice.

This solves the Service.ExternalName problems since the resource lifecycle can be limited by RBAC (related: https://github.com/kubernetes/kubernetes/issues/103675)

Unfortunately whether this is supported depends the DNS implementation.

EndpointSlice support was added in CoreDNS 1.8.1 (https://coredns.io/2021/01/20/coredns-1.8.1-release/) - which means K8s/kubeadm didn't pick up this change until K8s 1.22. We won't adopt K8s 1.22 as a minimum until next year so switch now would be a breaking change.

Current Approach

I think we should use Endpoints (2.1) even with the caveats mentioned and next year switch over to EndpointSlices (2.2) once 1.22 becomes our min K8s version. Then we can assume the CNAME record will work as expected.

Long term we should start the discussion with Gateway API folks to expose a filter type concept. Then DomainMapping could potentially bypass the Route's K8s service and hit the Revision's K8s service directly. It's currently unclear to me if that's an actual win since you're going through the kibe-proxy anyway. Probably worth an exploration.

Start by reviewing the Route and DomainMapping implementations, their use of Service.ExternalName, and the proposed approach in PoC PR #11747. Compare the Endpoints and EndpointSlices options and clarify the chosen design; done means DomainMapping continues to function without relying on ExternalName.