ClickHouse / ClickHouse/ClickHouse
Introspection (plan fragment dependencies, open telemetry, ...)
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
Extend the existing single-node telemetry/tracing infrastructure so that it also covers distributed execution, giving full end-to-end visibility into how a query plan is broken up and executed across the cluster.
This should include:
- Plan fragment dependencies: capturing which plan fragments are sent to which nodes, and how fragments depend on/relate to one another (parent/child relationships, data exchange between fragments).
- Cross-node trace propagation: ensuring the OpenTelemetry trace context (trace ID, span ID) initiated on the coordinator/initiator node is correctly propagated to all participating nodes, so spans from remote fragments attach to the same distributed trace rather than starting new, disconnected traces.
- Per-fragment spans: each node should emit spans for the fragment(s) it executes, mirroring the granularity already available for single-node execution (e.g., per-stage/per-operator spans), so a full trace shows both the "shape" of the distributed plan and per-node execution detail.
- Consistency with existing single-node model: reuse the current telemetry/tracing abstractions and conventions where possible, rather than introducing a parallel mechanism, so single-node and distributed traces are structurally consistent and can be visualized/analyzed the same way.
-
Current Plan:
- **[[done]](https://github.com/ClickHouse/ClickHouse/pull/114813)** PR 1: **Fix distributed trace propagation** by adding the OpenTelemetry context to the AsyncTaskExecutor Obj allowing OpenTelemetry to be part of the task execution like the ThreadPool behaviour, one telemetry span per AsyncTask. As a result, CLIENT span appears for distributed SELECTs and remote SERVER spans parent correctly under it.
- **[[done]](https://github.com/ClickHouse/ClickHouse/pull/115619)** PR 2: **Write sampled trace context back into ClientInfo** and enable trace id inside the client info when the query span is created.
- **[[doing]](https://github.com/ClickHouse/ClickHouse/pull/115784)** PR 3: **Per-shard spans on the classic distributed path**. Make the PR1 fiber holder the per-shard fragment span so RemoteQueryExecutor can attach clickhouse.cluster, clickhouse.shard_num, clickhouse.processed_stage, clickhouse.query_id, clickhouse.initial_query_id, clickhouse.target_host.
- PR 4: **Parallel replicas coverage: coordinator spans** Full coordinator coverage (PR1 already fixes propagation for replicas; PR3 gives per-replica connection spans and this PR will make sure the information from replicas are emitted and summarized.
- PR 5: **Trace propagation to stateless workers + worker task span** Make the make_distributed_plan worker path traced.
- PR 6: **Initiator-side fragment DAG spans.** Record the fragment DAG at execution time on the initiator. So a trace shows both the shape of the distributed plan and where each fragment ran.
- PR 7: **EXPLAIN DISTRIBUTED** A new ExplainKind that builds the fragment DAG without executing and renders stages, dependencies, exchanges, and task-to-host assignments. Introspection surface. After the run, it will include a bar representing the time spent on each stage, shards will be represented with indentation.
Contributor guide
Assessment
This issue has not been assessed yet.