need better facilities for post hoc debugging of slow API requests in production

A typical API request could spend time waiting on:

• input from the external client
• making a database query, which could be
  • for authentication
  • for authorization
  • arbitrary other stuff the request needs to do
• waiting for a shared resource, like
  • a database connection
  • a connection to another control plane service
• making a request to another control plane service
• waiting for some other control plane event to happen, like
  • a saga to complete

I'm sure there's more. Currently, we have very limited runtime visibility into a lot of this.

We currently use a variety of approaches here (sorry this is a crude summary):

• slog log entries: always on, easiest to get from prod systems today, easy to plumb correlation keys through. Currently missing a lot of coverage (though it's got the most of all of these) and hard to scale up to high-frequency events.
• DTrace probes: most powerful, but requires reproducibility, not much coverage today, and missing a lot of correlation keys. Potentially the hardest to operationalize when we can't log into systems directly. But we could do it, and in the limit it's probably necessary.
• Tracing data: currently not collectable in production and doesn't cover much of the above.
• Metrics: Doesn't currently cover much of the above and we haven't operationalized collecting it somewhere useful from production systems.

None of these covers very much of the events above. Slog is probably closest but it's missing a lot.

In terms of getting the data out: one approach I've seen success with is to accumulate the high-level events in a per-request buffer that gets included in the final slog log entry for the request. This winds up being something you can leave always-on and you can get rich detail after-the-fact just by grep'ing the Nexus log for the request id.

In terms of instrumenting the code: long ago we discussed an approach where we define our own macro for emitting events and the implementation of that can be adjusted by config (and maybe dynamically) to emit slog entries, fire DTrace probes, emit tracing entries, or update metrics. This would probably be a big migration lift but it'd make it possible for us to use a variety of tools and change tools without having to update the code instrumentation.

Start by surveying the existing slog entries, DTrace probes, tracing data, and metrics coverage described in the issue, since no files or tests are named. Define the supported request events and correlation data, then determine how completion would be validated across production logging and instrumentation paths.