open-telemetry / open-telemetry/opentelemetry-java
GRPC Exporter sends messages bigger than gRPC protocol allows
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2.5k
- Forks
- 1k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 58
Description
Describe the bug
I started getting the following error from OkHttpGrpcExporter in my logs:
Failed to export metrics. The request could not be executed. Full error message: stream was reset: NO_ERROR
Turned out, this happens because the exporter sends messages too big (default limit is 4mb). This is what I see on receiving side:
Header: grpc-message: grpc: received message larger than max (6117815 vs. 4194304)
Name Length: 12
Name: grpc-message
Value Length: 60
Value: grpc: received message larger than max (6117815 vs. 4194304)
[Unescaped: grpc: received message larger than max (6117815 vs. 4194304)]
Representation: Literal Header Field with Incremental Indexing - New Name
Steps to reproduce
I suppose I'm just sending a large number of metrics.
However, it might be reproducible by increasing the publishing interval (io.opentelemetry.sdk.metrics.export.PeriodicMetricReaderBuilder#setInterval(java.time.Duration)).
This will make the meter accumulate more data before sending.
What did you expect to see?
- I would expect the exporter to respect 4mb gRPC message limit and send more than one message in this case
- I would like to have a clearer error message about it. In this particular case, it would be useful to print response details (including headers)
What did you see instead?
The error message mentioned above, and it was not particularly helpful.
What version and what artifacts are you using?
io.opentelemetry:opentelemetry-api:1.18.0
io.opentelemetry:opentelemetry-context:1.18.0
io.opentelemetry:opentelemetry-exporter-common:1.18.0
io.opentelemetry:opentelemetry-exporter-otlp:1.18.0
io.opentelemetry:opentelemetry-exporter-otlp-common:1.18.0
io.opentelemetry:opentelemetry-sdk:1.18.0
io.opentelemetry:opentelemetry-sdk-common:1.18.0
io.opentelemetry:opentelemetry-sdk-logs:1.18.0-alpha
io.opentelemetry:opentelemetry-sdk-metrics:1.18.0
io.opentelemetry:opentelemetry-sdk-trace:1.18.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with OkHttpGrpcExporter and the PeriodicMetricReaderBuilder#setInterval reproduction described in the report. Verify behavior against the 4 MB gRPC limit and the oversized-metrics case; done means requests are split or bounded and export failures provide clearer response details.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- api, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100