Improve timeout configuration, esp for async operations

Today, we set gRPC timeouts on a per-service basis. So, for example, every request from the RA to the SA has the same timeout set, no matter whether that request is a read, a write, or a big batch operation; no matter whether that request is the only one in the whole code path or the first of many sequential requests; no matter whether an HTTP client is waiting on the result or we're doing async work.

For our most extreme example, take CAA checking. The RA calls VA.CheckCAA at two different times:

• during validation, which is async, and during which CAA is the last major operation (after DoDCV); and
• during finalization, which is synchronous, and during which CAA rechcking is one of the first major operations (before all issuance and getting SCTs).

Having the same timeouts across all methods on a service, and even across the same method called in different contexts, is not serving us well. We should have more granular control here.

Proposal 1: Keep everything in gRPC. The cmd.GRPCClientConfig would grow a new stanza which maps method names to specific timeouts; any unmapped methods would inherit the top-level timeout. This provides a uniform solution which can be applied across all of our gRPC clients as-needed, but doesn't handle the CheckCAA case. For cases like that, we could create a new RPC (e.g. va.RecheckCAA) which calls the same underlying server code but could have a separate timeout configured on the client side.

Proposal 2: Bespoke timeouts. Set all of our gRPC timeouts to some default value (say, 90s) which will serve only as a backstop. Configure the WFE with per-API-method timeouts, so that our normal "shave a few ms off at each gRPC layer" system can provide tighter deadlines for each component. Where we need custom tighter timeouts, e.g. for caa rechecking, add new bespoke config items and have the boulder code calling those methods manually call context.WithTimeout. This gives us fine-grained control over timeouts at all levels of the boulder, but doesn't form a holistic "system" for controlling timeouts.

In general, I lean towards Proposal 2. This is for two reasons: first, I think that adding gRPC methods like "va.RecheckCAA" solely for the sake of timeouts is unfortunate and unergonomic, and so Proposal 1 will end up having some custom timeouts like Proposal 2 anyway; and second, I like the idea of all of the WFE's API methods having timeouts because it makes it gives us a one-stop-shop to see how long these methods should actually be taking.

Start with cmd/config.go and grpc/interceptors.go to understand the current top-level gRPC timeout and deadline behavior, then trace the WFE API methods and the RA-to-VA CheckCAA calls. The issue presents two competing proposals rather than a decided implementation; done requires an agreed timeout design, configuration changes, and coverage for synchronous and asynchronous call contexts.