ClickHouse / ClickHouse/ClickHouse

Introspection (plan fragment dependencies, open telemetry, ...)

Open
#112,738 2 comments 1 reaction 1 assignee Claimed by @diegomestre2 View on GitHub
comp-logging
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

Extend the existing single-node telemetry/tracing infrastructure so that it also covers distributed execution, giving full end-to-end visibility into how a query plan is broken up and executed across the cluster.

This should include:

- Plan fragment dependencies: capturing which plan fragments are sent to which nodes, and how fragments depend on/relate to one another (parent/child relationships, data exchange between fragments).
- Cross-node trace propagation: ensuring the OpenTelemetry trace context (trace ID, span ID) initiated on the coordinator/initiator node is correctly propagated to all participating nodes, so spans from remote fragments attach to the same distributed trace rather than starting new, disconnected traces.
- Per-fragment spans: each node should emit spans for the fragment(s) it executes, mirroring the granularity already available for single-node execution (e.g., per-stage/per-operator spans), so a full trace shows both the "shape" of the distributed plan and per-node execution detail.
- Consistency with existing single-node model: reuse the current telemetry/tracing abstractions and conventions where possible, rather than introducing a parallel mechanism, so single-node and distributed traces are structurally consistent and can be visualized/analyzed the same way.
-
Current Plan:

- **[[done]](https://github.com/ClickHouse/ClickHouse/pull/114813)** PR 1: **Fix distributed trace propagation** by adding the OpenTelemetry context to the AsyncTaskExecutor Obj allowing OpenTelemetry to be part of the task execution like the ThreadPool behaviour, one telemetry span per AsyncTask. As a result, CLIENT span appears for distributed SELECTs and remote SERVER spans parent correctly under it.

- **[[done]](https://github.com/ClickHouse/ClickHouse/pull/115619)** PR 2: **Write sampled trace context back into ClientInfo** and enable trace id inside the client info when the query span is created.

- **[[doing]](https://github.com/ClickHouse/ClickHouse/pull/115784)** PR 3: **Per-shard spans on the classic distributed path**. Make the PR1 fiber holder the per-shard fragment span so RemoteQueryExecutor can attach clickhouse.cluster, clickhouse.shard_num, clickhouse.processed_stage, clickhouse.query_id, clickhouse.initial_query_id, clickhouse.target_host.

- PR 4: **Parallel replicas coverage: coordinator spans** Full coordinator coverage (PR1 already fixes propagation for replicas; PR3 gives per-replica connection spans and this PR will make sure the information from replicas are emitted and summarized.

- PR 5: **Trace propagation to stateless workers + worker task span** Make the make_distributed_plan worker path traced.

- PR 6: **Initiator-side fragment DAG spans.** Record the fragment DAG at execution time on the initiator. So a trace shows both the shape of the distributed plan and where each fragment ran.

- PR 7: **EXPLAIN DISTRIBUTED** A new ExplainKind that builds the fragment DAG without executing and renders stages, dependencies, exchanges, and task-to-host assignments. Introspection surface. After the run, it will include a bar representing the time spent on each stage, shards will be represented with indentation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.