open-telemetry / open-telemetry/opentelemetry-java
Add mechanism for exporters to signal backpressure to callers
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2.5k
- Forks
- 1k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 58
Description
Right now, our processors (BSP, BLRP, PMR) call exporters (SpanExporter, LogRecordExporter, MetricExporter) and wait for resulting CompletableResultCode to complete before calling again. This is meant to be protective of exporters: There's no guarantee that an exporter is written in such a way that it can handle the load of many concurrent calls to export. A well written exporter would respond immediately with CompletableResultCodes so long as it has extra concurrent processing capacity, and block calls to export when its capacity is full.
Since we can't ensure all exporters are written with this in mind, we take the conservative approach and do not call them concurrently.
But this limits the throughput of exporters that are written to be able to handle concurrent export calls while signalling backpressure to callers.
Let's adjust our exporter contracts so that they can signal to callers what types of concurrent / backpressure capabilities they have, such that BSP, BLRP, PMR can call export concurrently on exporters that advertise that capability.
This is related to https://github.com/open-telemetry/opentelemetry-java/issues/7099, https://github.com/open-telemetry/opentelemetry-java/issues/4264, but a specific implementation recommendation based improving exporters to be able to opt into concurrent invocation by BSP, BLRP, PMR.
cc @trask
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how BSP, BLRP, and PMR invoke SpanExporter, LogRecordExporter, and MetricExporter and wait on CompletableResultCode. Review the related issues #7099 and #4264 for context. Done means exporters can advertise concurrent or backpressure capabilities and the processors use that signal without weakening the existing protective behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100