open-telemetry / open-telemetry/opentelemetry-java
Optimize data loss issues for high-resource single-instance deployments that generate a large number of spans.
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2.5k
- Forks
- 1k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 58
Description
Is your feature request related to a problem? Please describe.
Some of our services are deployed as high-resource single instance. They often generate a large number of spans (sustaining up to approximately 15,000 spans per second, with peak bursts of 5,000 to 6,000 spans within 30ms).
The exporter seems to export Spans in only one thread, which limit the through out for sending. Also, it has a constant default value 2048 for queue size, which is too small for high-resource instances.
As a result, many Span were dropped because of full queue, which cause data loss.
Describe the solution you'd like
First, send data with multiple consumers. The default number of consumers can depends on the number of CPU cores, maybe half of cores is good.
Also, the default value of queue size should depend on the CPU cores or memory limit of the service.
Describe alternatives you've considered
Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the exporter queue and its single-consumer sending path, then inspect the related configuration and tests. The work is done when high-resource single-instance deployments can use multiple consumers and appropriately sized defaults without spans being dropped under the stated load.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability-sre, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100