`dtrace` for RPC traffic through linkerd2

Feature Request

dtrace exposes a query API that allows you to dynamically instrument program text in a running binary, to provide tracing information. So you can say something like "tell me any time a binary uses glibc's open".

It would be nice if linkerd2 provided the networking equivalent, exposing a query API that allowed you to dynamic attach a probe to the control plane that provides information about a distributed trace. So you could say: "give me a bunch of information about requests taking longer than x ms, aggregated across the multiple hops through all the microservices that the request touches."

I think this is useful because at scale it is useful to have principled ways of reasoning about an entire request rather than an aggregate metric. The ability of linkerd to introspect on a request over multiple RPC hops could be a critical step in this process.

What problem are you trying to solve?

ad hoc distributed tracing. I want the ability to understand the lifecycle of a request, beginning to end, dynamically.

How should the problem be solved?

Presumably with a query API that allows you to dynamically attach queries to the control plane. This could be dtrace-like, or ~relational, ReactiveX, or something else. It is worth considering exposing it as an SDK in a real language so that you can write queries that are like "tell pagerduty if query with x properties occurs".

Any alternatives you've considered?

There are distributed tracing systems out there, but they have the disadvantage that in many cases your code needs to emit metrics. So, if you're posting the wrong events, you have to change them and re-deploy the app.

Being able to dynamically instrument a global RPC control plane would be very convenient because it requires no changes to code.

How would users interact with this feature?

Users write a query somehow (e.g., through a CLI, in a text file), which is given linkerd2 to service.

No files, tests, or concrete entry points are named in the issue. Start by surveying Linkerd2's existing RPC tracing and control-plane interfaces; the issue is complete only once the query API and implementation scope are agreed and delivered.