container-cache: add a hermetic test harness with a request-counting fake origin

The cache has no test layer that can verify its own behaviour. Every existing suite either greps rendered templates or requires a live cluster, and the ones that reach the proxy do so with TLS verification disabled. As a result the properties the cache exists for are entirely unverified, and defects reach production that a modest harness would catch in seconds.

Current state

There is no hermetic layer: no local harness, no fake origin, no unit execution of the Lua.

Evidence this is the actual gap

Three container-cache defects found recently, and what caught each:

• Configurator wrote the wrong containerd plugin key on config schema v3 hosts. Caught quickly because that component is Python and has real unit tests.
• Minted leaf certificates omitted authorityKeyIdentifier, breaking every strict TLS client (#1180). Not caught: the only code that talks to the proxy over TLS sets verify=False, so no test could have observed it.
• service.type other than NodePort silently disabled cache routing. Only caught by adding a render assertion after the fact.

The component with a unit layer had its bug caught. The components without one did not.

Proposal

A hermetic harness built around a fake origin that counts requests. Every meaningful property of a cache is "how many times did you contact the origin", and that is unobservable against a real provider. The fake origin should mimic the provider contract: issue presigned URLs whose signature changes per request, return 302 with no-store, serve byte ranges, and expose a request counter.

That single capability makes the following testable, none of which is covered today:

• Two differently-signed URLs for one object produce exactly one origin fetch; the second request is a HIT.
• Rotating the authorization token still produces a HIT (isolation without cache fragmentation).
• N concurrent cold requests for one range produce exactly one origin fetch.
• A no-store redirect is cached and not re-fetched within its TTL.
• A 429 is never cached.
• A client with VERIFY_X509_STRICT completes the handshake (the #1180 regression class).

A second, cheaper layer: execute the Lua directly. cc-route.lua owner election is a pure function (md5(key) mod N, ordinal parsing, relay-marker short-circuit), and the SigV4 parameter filtering is pure string to string. Both are unit-testable with resty inside the existing proxy image.

Feasibility — already verified

The main risk was whether the proxy runs outside Kubernetes. It does. Confirmed today:

• nvcf-container-cache and nvcf-proxy-tls-certs pull normally.
• Full proxy configuration extracts from helm template output into a directory tree.
• Signing CA material is reproducible by replicating the StatefulSet init container non-Vault path (copy intermediate.key.pem to leaf-ca.key, concatenate the chain into ssl-bundle.crt).
• The proxy container then starts cleanly with only the benign user directive warning also seen in production, and mints per-SNI leaves on request.

So the harness needs: a compose file, the fake origin, hostname overrides pointing the proxy at it, and the assertions. No cluster, no credentials, no GPU.

Suggested sequencing

1. Fake origin plus compose harness, with the six assertions above.
2. Fix verify=False in the existing e2e suites so they stop masking certificate faults, and demote them from correctness to smoke.
3. Lua unit layer via resty.

References

Relates to #1180