Kafka Connect: Add OpenTelemetry distributed tracing support
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
### Summary
Add optional OpenTelemetry (OTEL) distributed tracing to the Iceberg Kafka Connect sink connector. When enabled, the connector should create spans for the two main pipeline stages and continue upstream traces by extracting the W3C `traceparent` header from Kafka Connect record headers.
This is **span tracing** for the Connect sink pipeline — distinct from existing OTEL **metrics** work in the core library ([#16169](https://github.com/apache/iceberg/issues/16169)) and JMX **metrics** for the commit pipeline ([#17025](https://github.com/apache/iceberg/pull/17025)).
### Motivation / use case
End-to-end CDC pipelines commonly look like:
**Source DB → Debezium (or similar) → Kafka → Iceberg Kafka Connect sink → Iceberg tables**
Debezium already supports distributed tracing via `ActivateTracingSpan` (standard Connect sources) and connector-native tracing (e.g. [debezium-connector-cassandra#173](https://github.com/debezium/debezium-connector-cassandra/pull/173)), injecting `traceparent` into Kafka/Connect record headers. Today the Iceberg sink does not consume that context, so traces break at the sink boundary and operators cannot answer:
- What is the lag from source CDC event to Iceberg ingest?
- What is the lag from ingest (buffer) to catalog commit (durability)?
- Which records/commits failed, and with what error?
The Iceberg Kafka Connect sink also has a **two-phase** write path (buffer to Parquet/ORC files on `put()`, then async coordinator commit to the catalog). Without connector-level spans, generic worker instrumentation alone cannot distinguish ingest latency from commit latency.
### Query engine
None
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start by locating the Iceberg Kafka Connect sink implementation's put() path and async coordinator commit path, then inspect how Kafka Connect record headers expose W3C traceparent. Done means tracing is optional, spans cover both pipeline stages, upstream context continues, and focused connector tests cover the enabled and disabled paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- distributed-systems, observability, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100