open-telemetry / open-telemetry/opentelemetry-java

GRPC Exporter sends messages bigger than gRPC protocol allows

Open
#5,394 14 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug
Dominant language
Java
Stars
2.5k
Forks
1k
Avg merge
3d 17h
Merged PRs (30d)
58

Description

Describe the bug
I started getting the following error from OkHttpGrpcExporter in my logs:

Failed to export metrics. The request could not be executed. Full error message: stream was reset: NO_ERROR

Turned out, this happens because the exporter sends messages too big (default limit is 4mb). This is what I see on receiving side:

Header: grpc-message: grpc: received message larger than max (6117815 vs. 4194304)
    Name Length: 12
    Name: grpc-message
    Value Length: 60
    Value: grpc: received message larger than max (6117815 vs. 4194304)
    [Unescaped: grpc: received message larger than max (6117815 vs. 4194304)]
    Representation: Literal Header Field with Incremental Indexing - New Name

Steps to reproduce
I suppose I'm just sending a large number of metrics.

However, it might be reproducible by increasing the publishing interval (io.opentelemetry.sdk.metrics.export.PeriodicMetricReaderBuilder#setInterval(java.time.Duration)).
This will make the meter accumulate more data before sending.

What did you expect to see?

  1. I would expect the exporter to respect 4mb gRPC message limit and send more than one message in this case
  2. I would like to have a clearer error message about it. In this particular case, it would be useful to print response details (including headers)

What did you see instead?
The error message mentioned above, and it was not particularly helpful.

What version and what artifacts are you using?

io.opentelemetry:opentelemetry-api:1.18.0
io.opentelemetry:opentelemetry-context:1.18.0
io.opentelemetry:opentelemetry-exporter-common:1.18.0
io.opentelemetry:opentelemetry-exporter-otlp:1.18.0
io.opentelemetry:opentelemetry-exporter-otlp-common:1.18.0
io.opentelemetry:opentelemetry-sdk:1.18.0
io.opentelemetry:opentelemetry-sdk-common:1.18.0
io.opentelemetry:opentelemetry-sdk-logs:1.18.0-alpha
io.opentelemetry:opentelemetry-sdk-metrics:1.18.0
io.opentelemetry:opentelemetry-sdk-trace:1.18.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with OkHttpGrpcExporter and the PeriodicMetricReaderBuilder#setInterval reproduction described in the report. Verify behavior against the 4 MB gRPC limit and the oversized-metrics case; done means requests are split or bounded and export failures provide clearer response details.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
api, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.