spike(container-cache): redirect to the owner pod instead of relaying, to remove the network doubling

Summary

Under consistent-hash routing the cache tier relays roughly (N-1)/N of requests
to the owner pod, and the relayed body crosses the network twice. This tracks
the alternative: answer with a redirect to the owner so the body crosses once.

The design is not settled. Two of the open risks could invalidate it, so this is
a tracking issue for a spike, not a plan of record.

Why

With N pods and clients arriving uniformly through the Service, (N-1)/N of
requests are not local. The hash key is the per-range cache key, so a single
large object's byte ranges deliberately spread across all owners and one client
downloading one object relays most of its chunks.

Every relayed byte crosses the network twice, owner to relay pod to client, so
per pod NIC traffic is about 2.3x what it would be without the hop on a 3 pod
tier. Measured over roughly 41 hours on one production tier: about 520 TB served
to clients, 99.98 percent hit rate, average object about 528 MB.

Two cheaper options were considered and rejected:

• Caching on the relay. Removes the hop for hot objects, but only by writing a
  second copy of them to disk, which breaks the single-copy property the hash
  exists to provide. Rejected, and reverted from the connection-reuse PR.
• Replication factor of two in owner selection. Halves the relay fraction, but
  has the same second-copy cost.

A redirect is the only option that reduces the network doubling without putting
a second copy on disk.

Shape

Today:

client -> pod A (not owner)
          pod A -> pod B (owner)   [body]
          pod B -> pod A           [body]
          pod A -> client          [body]

Proposed:

client -> pod A:  GET /path            Range: bytes=...
pod A -> client:  307 Location: https://<peer-B>:<port>/path
client -> pod B:  GET /path            (Range re-sent)
pod B -> client:  [body]

Pod A answers with a header and leaves the data path. Cost is one extra round
trip against a multi-hundred-MB transfer.

It also removes a failure mode. On a miss today, proxy_cache_lock means the
owner emits no response header until the fill completes, so a relayed client
gets no first byte until the whole object has been fetched. That is why
relayReadTimeout is 300s. With a redirect the client waits on the owner
directly and that timeout stops existing.

Origin identity

The owner has to know which upstream to fetch from on a miss, and the redirect
changes the Host to the peer's name. Upstream selection today is purely
server_name matching:

server_name ~^x?l?files\.(stg\.)?ngc\.nvidia\.com$;
server_name huggingface.co;
server_name ~^(.+\.)?hf\.co$;

A request arriving with a peer hostname matches none of them and falls into
default_server, which is the S3 block.

Adding ?ns=<origin> to the redirect is the obvious fix and should be avoided.
These are presigned URLs whose signature covers the query string, so the owner
would have to strip the parameter before proxying upstream. A mistake there
surfaces as an intermittent 403 that looks like an auth failure. It also puts
Lua query-string surgery on the code path carrying Signature, ssec-key and
versionId.

Preferred instead: encode the origin in the listener port, one peer port per
origin, with the target host fixed per block. This is the pattern the CRI-O path
already uses, for the same reason (CRI-O cannot pass ?ns=). The query string
then passes through byte for byte and signatures are untouched.

Note the cache key is unaffected either way, since it is built from $uri,
which excludes the query string.

Open risks

Ordered by how likely they are to invalidate the approach.

1. Authorization stripping. The client on the model path is python-requests,
   which strips the Authorization header on a cross-host redirect. The proxy's
   auth model is a per-resource probe using exactly that header, and a request
   arriving without it is treated as anonymous and bypasses the cache. That
   would turn a bandwidth optimisation into a cache-disabling change. It may not
   bite where URLs are presigned and carry credentials in the query, but that
   needs proving per path and per client.

2. Egress policy. Tenant pods currently reach the cache through a single
   ClusterIP, which is referenced in egress allow-lists. A redirect requires
   them to reach per-pod peer Services directly.

3. TLS validation. Certificates are minted per SNI and signed by the same CA,
   and workload pods already carry that CA bundle, so this should work. Two
   details are unverified: peer service DNS names are long enough to trip the
   CN length limit, which substitutes a wildcard for the first label, and no
   client has been tested against a minted peer certificate.

4. Availability failover. Today this is server-side and invisible to the client
   (proxy_next_upstream error timeout falls to the backup ordinal before the
   response starts). With a redirect, if the owner is down the client gets a
   connection error and nothing retries on its behalf. Health detection is
   passive (max_fails / fail_timeout), so the redirecting pod only knows
   after failures.

5. Information disclosure. The redirect hands internal peer service names to
   tenant workloads and requires them to connect to specific cache pods. Worth a
   security review, not only a design one.

Test matrix

Area	What has to be proven
TLS	peer serves a valid chain for its own name; CN wildcard substitution does not break validation; real clients accept it
Auth	Authorization survives the host change, or is provably unnecessary, per client and per path
Range	307 preserves Range; partial content still correct across the redirect
Origin	port-per-origin selects the right upstream on a miss; presigned query passes through unmodified
Failover	owner down at redirect time, and owner down mid-transfer
Egress	tenant pods can reach per-pod peer Services under current policy
Clients	the NGC and HuggingFace downloaders, at deployed versions

Cheapest first step is the TLS row: curl a peer name against a minted
certificate. It needs no cluster change and answers risk 3 on its own.

Prerequisite

Land the connection-reuse and observability work first (#1037) and read the
route label for a day. The (N-1)/N relay fraction is derived from the hash and
uniform arrival, not measured. If reusing connections flattens the latency, this
may not be worth building; if it does not, the label quantifies exactly what
there is to gain.

Not in scope

Relay-side caching and replication factor two, both rejected above for putting a
second copy on disk.

Start with the TLS row in the test matrix: use curl against a peer name and a minted certificate, without a cluster change. Read prerequisite #1037 and the route label results before evaluating the remaining auth, range, origin, failover, egress, and client checks; done means the risks are proven safe or the redirect approach is rejected.