open-telemetry / open-telemetry/opentelemetry-java

Optimize data loss issues for high-resource single-instance deployments that generate a large number of spans.

Open
#7,508 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Feature Request help wanted
Dominant language
Java
Stars
2.5k
Forks
1k
Avg merge
3d 17h
Merged PRs (30d)
58

Description

Is your feature request related to a problem? Please describe.
Some of our services are deployed as high-resource single instance. They often generate a large number of spans (sustaining up to approximately 15,000 spans per second, with peak bursts of 5,000 to 6,000 spans within 30ms).

The exporter seems to export Spans in only one thread, which limit the through out for sending. Also, it has a constant default value 2048 for queue size, which is too small for high-resource instances.

As a result, many Span were dropped because of full queue, which cause data loss.

Describe the solution you'd like
First, send data with multiple consumers. The default number of consumers can depends on the number of CPU cores, maybe half of cores is good.

Also, the default value of queue size should depend on the CPU cores or memory limit of the service.

Describe alternatives you've considered

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the exporter queue and its single-consumer sending path, then inspect the related configuration and tests. The work is done when high-resource single-instance deployments can use multiple consumers and appropriately sized defaults without spans being dropped under the stated load.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability-sre, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.