Domain Mapping internal linkage over http causes issues with certain Oauth redirect flows

What version of Knative?

Seen on Knative Serving 1.6

Trying to reproduce on main

Scenario

A java app deployed as knative service which redirects to a Spring Security SSO endpoint.

A domain mapping is configured to point to that Knative service and use https with a custom secret.

The app itself is a Spring Boot app leveraging Spring Security. The profile we're using redirects to a self-hosted Keycloak instance as a SSO provider.

Expected Behavior

https is used throughout the entire auth redirect flow, and you can log in successfully.

In more detail, when hitting an https endpoint configured via DomainMapping, we would expect the workload (pod) to receive traffic with x-forwarded-proto: https correctly set.

When using auto-tls, this scenario works.

Actual Behavior

Somewhere in the auth redirect flow, the redirect URI becomes HTTP. Thus, when the auth endpoint tries to redirect back to the knative service with the cookies set and the "secure" flag set, it redirects to an http endpoint and the connection fails.

What we think is going on

We learned that when you use a domain mapping, it is telling the request to get forwarded to the app.svc.cluster.local domain for the Ksvc, and sticking the domainmapping url in a header.

The problem we ran into is that the connection to app.svc.cluster.local (basically looping back to envoy) is done via http. So when the oauth part of the request flow must send a redirect link back to the ksvc, the protocol is http set in the x-forwarded-proto header.

This means that the ksvc calls Spring Security's SSO endpoint over http (and we saw the X-Forwarded-Proto: http in that case). Ultimately this generates a redirect_uri with an http protocol.

The end result is the SSO provider is spitting back a 302 redirect with a bunch of set-cookie headers with the location header set to http://<FQDN> instead of https://<FQDN> and includes the Secure flag. The redirect is to an insecure endpoint, thus the cookies don't atttach and SSO fails (https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#creating_cookies).

When done with auto-tls on the other hand, there is no loop back to the cluster local domain, and tit maintains the X-Forwarded-Proto: https header when it hits Spring Security's SSO endpoint. This correctly generates a redirect_uri with an https protocol.

We tried to draw this out here:

Steps to Reproduce the Problem

Working on creating a pared down version of this scenario that can be used as an example.

You can see the behavior though if you deploy a simple ksvc that says hello and dumps the request data.

I've deployed a ksvc hello and configured auto-tls. When I reach my app, I see:

$ curl -k https://hello.default.kauz.tanzu.biz/
Hello folks!
Request: GET / HTTP/1.1
Host: hello.default.kauz.tanzu.biz
Accept: */*
Forwarded: for=10.24.2.1;proto=https, for=10.24.2.246
K-Proxy-Request: activator
User-Agent: curl/7.79.1
X-Envoy-Internal: true
X-Forwarded-For: 10.24.2.1, 10.24.2.246, 10.24.1.124
X-Forwarded-Proto: https
X-Request-Id: 326b8740-0589-47c8-b8b2-3a1ecd74082b
X-Request-Start: t=1671472237.408

When I create a domain mapping for this svc and hit my app, you can see that X-Forwarded-Proto: http.

$ curl -k https://hello.clay.tanzu.biz/
Hello folks!
Request: GET / HTTP/1.1
Host: hello.clay.tanzu.biz
Accept: */*
Forwarded: for=10.24.2.1;proto=http, for=10.24.2.246, for=10.24.1.120
K-Proxy-Request: activator
User-Agent: curl/7.79.1
X-Envoy-External-Address: 10.24.2.246
X-Forwarded-For: 10.24.2.1,10.24.2.246, 10.24.1.120, 10.24.1.124
X-Forwarded-Proto: http
X-Request-Id: 11a8b53f-159a-4751-95f6-f2d5b245bc54
X-Request-Start: t=1671472246.941
X-Request-Start: t=1671472246.943

I think this is Envoy doing what it supposed to do. When Spring makes the redirect URI based on these headers, see here), it makes a problem for the auth flow.

Start by reproducing the domain-mapping request and compare its forwarded headers with the auto-TLS request, focusing on X-Forwarded-Proto and the OAuth redirect flow. Use the reported domain-mapping and Envoy behavior as entry points; done means HTTPS remains represented throughout the flow so the redirect URI and secure cookies work.