feat(connectors): OTLP/gRPC source connector for logs, metrics and traces
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
### Description
Getting OpenTelemetry data into Iggy currently means running a separate bridge
process in front of it. Collectors and SDKs already speak OTLP/gRPC, and the
connectors runtime already hosts long-lived plugins, so the bridge is avoidable:
the OTLP service can terminate inside the runtime and produce straight onto Iggy
topics.
OTLP is listed as a P0 source and sink on the connector roadmap (#2753).
### Affected area / component
Connectors
### Proposed solution
An `otlp_source` connector plugin that binds the OTLP standard port and serves
`LogsService`, `MetricsService` and `TraceService` from a single listener, since
standard OTLP deployments expose one endpoint and splitting them would force
operators to configure three addresses that match no existing tooling.
Decode with `opentelemetry-proto`, which already ships the generated tonic
stubs, so no `build.rs` or `protoc` enters the build.
Two storage formats:
- `json`, one queryable document per record, for sinks that need field-level
access.
- `proto`, the raw request bytes as a single message, roughly 4-5x smaller and
zero-copy when paired with an OTLP sink.
Backpressure must not fail the export. When the in-process channel fills, enqueue
what fits and report the remainder in the response's `partial_success`, counted
in OTLP records rather than internal messages, so a client retries only what was
actually rejected instead of resending a batch that was half delivered.
### Alternatives considered
Running the OpenTelemetry Collector with a custom exporter. That keeps a second
process in the path and duplicates the connector runtime's lifecycle, config and
metrics handling.
Generating the protobuf stubs with `tonic-build`. It adds `protoc` to the build
for types `opentelemetry-proto` already publishes.
### Contribution
Yes, I am willing to submit a PR.
Contributor guide
Research direction
Start by locating the connectors runtime and how its long-lived plugins are configured, then inspect the opentelemetry-proto generated tonic stubs for LogsService, MetricsService, and TraceService. The work is done when one otlp_source listener supports the json and proto formats and reports record-level partial_success under channel backpressure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, rust
- Domain
- backend, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100