apache / apache/iggy

feat(connectors): OTLP/gRPC source connector for logs, metrics and traces

Open
#3,815 1 comment 0 reactions 0 assignees View on GitHub
connectors
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Description

Getting OpenTelemetry data into Iggy currently means running a separate bridge
process in front of it. Collectors and SDKs already speak OTLP/gRPC, and the
connectors runtime already hosts long-lived plugins, so the bridge is avoidable:
the OTLP service can terminate inside the runtime and produce straight onto Iggy
topics.

OTLP is listed as a P0 source and sink on the connector roadmap (#2753).

### Affected area / component

Connectors

### Proposed solution

An `otlp_source` connector plugin that binds the OTLP standard port and serves
`LogsService`, `MetricsService` and `TraceService` from a single listener, since
standard OTLP deployments expose one endpoint and splitting them would force
operators to configure three addresses that match no existing tooling.

Decode with `opentelemetry-proto`, which already ships the generated tonic
stubs, so no `build.rs` or `protoc` enters the build.

Two storage formats:

- `json`, one queryable document per record, for sinks that need field-level
access.
- `proto`, the raw request bytes as a single message, roughly 4-5x smaller and
zero-copy when paired with an OTLP sink.

Backpressure must not fail the export. When the in-process channel fills, enqueue
what fits and report the remainder in the response's `partial_success`, counted
in OTLP records rather than internal messages, so a client retries only what was
actually rejected instead of resending a batch that was half delivered.

### Alternatives considered

Running the OpenTelemetry Collector with a custom exporter. That keeps a second
process in the path and duplicates the connector runtime's lifecycle, config and
metrics handling.

Generating the protobuf stubs with `tonic-build`. It adds `protoc` to the build
for types `opentelemetry-proto` already publishes.

### Contribution

Yes, I am willing to submit a PR.

Contributor guide

Open the contributing guide

Research direction

Start by locating the connectors runtime and how its long-lived plugins are configured, then inspect the opentelemetry-proto generated tonic stubs for LogsService, MetricsService, and TraceService. The work is done when one otlp_source listener supports the json and proto formats and reports record-level partial_success under channel backpressure.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, rust
Domain
backend, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.