Istio sidecars flood CoreDNS because of ExternalName services with ports that Knative creates for inference services

What version of Knative?

0.23.3

Summary

KNative creates ExternalName services for each inference service for redirecting traffic to Istio IngressGateway. For each such service, all Istio sidecars, every 5 seconds, will try to resolve the specified DNS target. As a result CoreDNS gets flooded. This is even worse on EKS where you have ndots: 5 and ec2.internal in search domains, that is, each DNS query results to five with one of them getting forwarded to AWS nameservers outside the cluster.

Steps to Reproduce the Problem

1. We use Knative Serving 0.23.3 with Istio 1.9.6 and KFServing 0.6.1.
2. We create an InferenceService with a transformer and a predictor.
3. KFServing creates an ExternalName service without ports configuration. In our setup it points to knative-serving-cluster-ingressgateway.knative-serving.svc.cluster.local.
4. KNative creates two ExternalName services with ports configuration (see https://github.com/knative/serving/commit/09986741f0bc6e369ed99370728ad715054656d5). In our setup it points to knative-serving-cluster-ingressgateway.knative-serving.svc.cluster.local.
5. Istio will create a STRICT_DNS cluster only for ExternalName services with ports configuration (see also https://github.com/istio/istio/issues/23463, https://github.com/istio/istio/issues/37331).
6. All Istio sidecars (envoy) running in the cluster, for each STRICT_DNS cluster, they will try to resolve the specified DNS target every 5 seconds (see https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/service_discovery#strict-dns). In our setup this is knative-serving-cluster-ingressgateway.knative-serving.svc.cluster.local.
7. On EKS because of ndots: 5 this will result in 5 DNS requests (4 NXDOMAIN for each search domains and one NOERROR) (see also https://discuss.istio.io/t/flood-of-nxdomain-lookups-to-coredns-from-istio-sidecar/11588)
8. On EKS, because the last seach domain is ec2.internal (on us-east-1), one of the above DNS requests will be forwarded to AWS nameservers that will respond with NXDOMAIN.

In a cluster with lots of pods with Istio sidecars and lots of inference services CoreDNS gets flooded with DNS requests. We have seen it going into CrashLoopBackoff and getting i/o timeout when talking to AWS nameservers:

[ERROR]: plugins/error 2 knative-serving-cluster-ingressgateway.knative-serving.svc.cluster.local.ec2.internal. A: read udp 10.52.76.80:XXX -> 10.52.0.2:53: i/o timeout

Start with the ExternalName service creation introduced by commit 09986741f0bc6e369ed99370728ad715054656d5 and reproduce the inference-service setup described above. Trace how ports are added to the generated services, then verify the result against Istio's STRICT_DNS behavior and CoreDNS query load; done should prevent the reported DNS flooding without breaking traffic redirection.