open-telemetry / open-telemetry/opentelemetry-cpp

[SDK] Support non-exporting pipeline log record processors (needed for EventToSpanEventBridgeProcessor)

Open
#4,454 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

discuss triage/accepted
Dominant language
C++
Stars
1.4k
Forks
632
Avg merge
1d 13h
Merged PRs (30d)
75

Description

Summary

EventToSpanEventBridgeProcessor (proposed in #4309) bridges log-record events onto the live Span referenced by the log record's resolved context. It needs a LogRecordProcessor that can:

  • read the resolved context (Context/live Span) at OnEmit() time
  • read the log record's own data (body, attributes) without taking exclusive ownership away from downstream processors/exporters

The current LogRecordProcessor interface doesn't support this shape of "non-exporting", read/mutate pipeline processor, so the bridge processor can't be implemented correctly without new SDK architecture.

Background

From review on #4309:

Processors like the bridge processor will require some new architectural components and changes to existing components to be spec compliant and maintain reasonable performance. This is in addition to the Logger API level EmitLogRecordWithContext change, which is needed.

The architecture components / changes that seem to be required include:

  1. Standardize the SDK built-in exporters (OTLP, ostream) on a recordable with a read-write interface. Pipeline processors need to read data from the record and not duplicate it.
  2. Create a new processor interface for composable pipeline processors that can read and mutate data but do not own the recordable life-cycle.
  3. Create a new root level processor that supports pipeline processors
  4. Update the LoggerContext to support setting a new root processor.

The concrete problem: with today's LogRecordProcessor interface, a configuration like

logger_provider:
  processors:
    my_logs_filter:
    my_log_record_sanitizer:
    event_to_span_bridge:
    batch:
      exporter:
        otlp_http:

fans out via MultiLogRecordProcessor instead of forming a pipeline: each processor gets an independent copy of the record via OnEmit(), which transfers ownership. That means N copies of the record (N times the recording cost) and the exporter at the end receives the raw, unfiltered/unsanitized record rather than the output of the upstream stages, since mutations made by an earlier stage are made on a copy that is then discarded.

See the full discussion: https://github.com/open-telemetry/opentelemetry-cpp/pull/4309#pullrequestreview-4953987049 and https://github.com/open-telemetry/opentelemetry-cpp/pull/4309#discussion_r3798294882

Scope

  • Standardize SDK built-in exporters (OTLP, ostream) on a ReadWriteLogRecord-style recordable.
  • New processor interface for composable pipeline processors that mutate data in place without owning the recordable lifecycle.
  • New root-level processor implementing a pipeline of these composable processors.
  • LoggerContext support for installing this new root processor.
  • Once the above lands, implement EventToSpanEventBridgeProcessor against the new interface and reintroduce it into declarative configuration (currently the config model/parser accepts the event_to_span_bridge processor block, but SdkBuilder only logs a warning that it is not yet supported; see the FIXME-SDK comment in sdk_builder.cc).

Status

Pending design and implementation of the components above. #4309 was scoped down to just the configuration model and YAML parser per this discussion, with the actual processor implementation deferred until this issue is resolved.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing LogRecordProcessor and MultiLogRecordProcessor design, then inspect LoggerContext and the FIXME-SDK comment in sdk_builder.cc. Define the required ReadWriteLogRecord and composable pipeline components, root processor, and LoggerContext integration before implementing EventToSpanEventBridgeProcessor. Done means the architecture works without transferring record ownership and the bridge can be reintroduced into declarative configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.