apache / apache/iceberg

Kafka Connect: Add OpenTelemetry distributed tracing support

Open
#17,115 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

### Summary

Add optional OpenTelemetry (OTEL) distributed tracing to the Iceberg Kafka Connect sink connector. When enabled, the connector should create spans for the two main pipeline stages and continue upstream traces by extracting the W3C `traceparent` header from Kafka Connect record headers.

This is **span tracing** for the Connect sink pipeline — distinct from existing OTEL **metrics** work in the core library ([#16169](https://github.com/apache/iceberg/issues/16169)) and JMX **metrics** for the commit pipeline ([#17025](https://github.com/apache/iceberg/pull/17025)).

### Motivation / use case

End-to-end CDC pipelines commonly look like:

**Source DB → Debezium (or similar) → Kafka → Iceberg Kafka Connect sink → Iceberg tables**

Debezium already supports distributed tracing via `ActivateTracingSpan` (standard Connect sources) and connector-native tracing (e.g. [debezium-connector-cassandra#173](https://github.com/debezium/debezium-connector-cassandra/pull/173)), injecting `traceparent` into Kafka/Connect record headers. Today the Iceberg sink does not consume that context, so traces break at the sink boundary and operators cannot answer:

- What is the lag from source CDC event to Iceberg ingest?
- What is the lag from ingest (buffer) to catalog commit (durability)?
- Which records/commits failed, and with what error?

The Iceberg Kafka Connect sink also has a **two-phase** write path (buffer to Parquet/ORC files on `put()`, then async coordinator commit to the catalog). Without connector-level spans, generic worker instrumentation alone cannot distinguish ingest latency from commit latency.

### Query engine

None

### Willingness to contribute

- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start by locating the Iceberg Kafka Connect sink implementation's put() path and async coordinator commit path, then inspect how Kafka Connect record headers expose W3C traceparent. Done means tracing is optional, spans cover both pipeline stages, upstream context continues, and focused connector tests cover the enabled and disabled paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
distributed-systems, observability, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.