open-telemetry / open-telemetry/opentelemetry-java

Add mechanism for exporters to signal backpressure to callers

Open
#8,831 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Feature Request
Dominant language
Java
Stars
2.5k
Forks
1k
Avg merge
3d 17h
Merged PRs (30d)
58

Description

Right now, our processors (BSP, BLRP, PMR) call exporters (SpanExporter, LogRecordExporter, MetricExporter) and wait for resulting CompletableResultCode to complete before calling again. This is meant to be protective of exporters: There's no guarantee that an exporter is written in such a way that it can handle the load of many concurrent calls to export. A well written exporter would respond immediately with CompletableResultCodes so long as it has extra concurrent processing capacity, and block calls to export when its capacity is full.

Since we can't ensure all exporters are written with this in mind, we take the conservative approach and do not call them concurrently.

But this limits the throughput of exporters that are written to be able to handle concurrent export calls while signalling backpressure to callers.

Let's adjust our exporter contracts so that they can signal to callers what types of concurrent / backpressure capabilities they have, such that BSP, BLRP, PMR can call export concurrently on exporters that advertise that capability.

This is related to https://github.com/open-telemetry/opentelemetry-java/issues/7099, https://github.com/open-telemetry/opentelemetry-java/issues/4264, but a specific implementation recommendation based improving exporters to be able to opt into concurrent invocation by BSP, BLRP, PMR.

cc @trask

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how BSP, BLRP, and PMR invoke SpanExporter, LogRecordExporter, and MetricExporter and wait on CompletableResultCode. Review the related issues #7099 and #4264 for context. Done means exporters can advertise concurrent or backpressure capabilities and the processors use that signal without weakening the existing protective behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.