Envoy: support periodic re-authentication of persistent connections

Currently persistent connections are only authorized when first established. This means that for applications using a connection pool, they may still be able to access resources for some time after a DENY intention has been added unless the destination service's proxies have been restarted. While that is possible as a break-glass solution, it would be ideal to have Envoy able to drop connections as soon as they loose authorization.

This is really a placeholder ticket since the bulk of the work here needs to be contributed to Envoy and an issue will be opened there referencing this one when we are ready to make a firm proposal and solicit feedback on it.

There are a few different theoretical options that solve the problem in decreasingly ideal ways but likely increasing simplicity/chance of being accepted upstream:

1. The holy grail would be to introduce a whole new streaming version of the ExtAuthZ gRPC service protocol which could efficiently manage tracking many thousands of connections at once and would be notified if any existing connections authorization has been revoked. This would allow virtually instant response to changes in policy but has a lot of complexity and potential performance overhead to implement let alone agree with rest of Envoy Community that it's a good idea. For now we won't pursue this unless others are interested.

2. Next best would be to extend the current ext_authz network extension to allow configuring a periodic re-auth timer. This would run per-connection and when it times out, the same Check RPC would be called as in a new connection. If it succeeds then the connection is left alone but if it fails the connection would be closed. This has a tradeoff - more frequent checks puts load on the AuthZ service as well as timer bookkeeping and RPC overhead in Envoy - when handling thousands of connections this could be significant. Less frequent checks would increase the window for stolen credentials or changes in policy to take effect for existing connections. If that window was in the order of a few minutes that seems OK in practice although still not ideal for demoing Intentions taking effect!
2.a. A nice optimization here that may be worthwhile would be to "single flight" and cache AuthZ responses per service identity. This would mean that if there are only a few client services, that means only a few AuthZ requests per time interval even with thousands of connections open. One downside would be loosing the ability to blacklist specific certificate serials. We could de-dupe by serial/sha instead which would mean that we do only one AuthZ per interval per distinct client cert even if that client is holding open 100 connections. Likely still a decent win if it's feasible in Envoy.

3. The simplest thing that would improve upon what we have but be simpler to implement would be to add a simple "max connection lifetime" config for Envoy ext_authz or tcp_proxy extensions. When this time is up the proxy will just close the connection regardless of idleness or otherwise. This is not ideal as it may interrupt service traffic unnecessarily, but would be simpler to implement most likely.

First part of this issue is to do some closer exploration in the Envoy code and see what is going to make most sense to propose there. My feeling is we need to shoot for at least 2 and possibly 1 if we can find a simple enough way to achieve it and others agree with the design. Previosuly discussed 2 with Envoy authors informally and they seemed to agree it was a reasonable option.

Start by exploring Envoy's ext_authz and tcp_proxy extensions, focusing on how Check RPCs and connection lifetimes are handled. Compare periodic re-authentication, response caching or single-flight, and maximum connection lifetime, then document a concrete proposal for a follow-up Envoy issue.

Written by the indexing model from the issue text.