elastic / elastic/apm-server

OTLP/gRPC message size limit is fixed at 4MB

Open
#14,759 3 comments 4 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

If the decompressed grpc message exceeds 4MB, the collector sending to apm-server logs the following error.
```
2024-11-25T04:08:26.561Z error internal/queue_sender.go:100 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = ResourceExhausted desc = grpc: received message after decompression larger than max (6624956 vs. 4194304)", "dropped_items": 11613}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
go.opentelemetry.io/collector/exporter@v0.110.0/exporterhelper/internal/queue_sender.go:100
```

The limit comes from the default 4MB grpc server message size limit `MaxRecvMsgSize` in https://pkg.go.dev/google.golang.org/grpc#MaxRecvMsgSize .

## Current user workarounds
- if batchprocessor is used, adjust the config values to influence the payload size
- use otlphttpexporter in place of otlpexporter in the collector sending to apm-server.

## Proposed solution
apm-server should either document or adjust the limit, or make it configurable.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.