opensearch-project / opensearch-project/data-prepper
[BUG] StatusCode.UNAVAILABLE grpc response when reaching armeria connection limit
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
The OTel trace source is configured with a connection limit (max_connection_count) which defaults to 500.
In case this limit is reached Data Prepper logs sth. like:
[armeria-boss-http-*:21890] WARN com.linecorp.armeria.server.ConnectionLimitingHandler - Dropped 10 connection(s) to limit the number of open connections to 500
and returns a StatusCode.UNKNOWN:
<_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "Received http2 header with status: 500"
debug_error_string = "UNKNOWN:Error received from peer {grpc_status:2, grpc_message:"Received http2 header with status: 500"}"
This status code is not classified as retryable failure.
Since clients are dropping the connection in this case, I am wondering if Data Prepper should respond with e.g. StatusCode.UNAVAILABLE instead so that this could be retried within the OTel client.
However, if a client consistently reaches the connection limit, retries could again fail of course.
To Reproduce
Steps to reproduce the behavior:
- Configure data prepper with the trace analytics setup.
- Reduce the
max_connection_countto a lower number so that this is reproducible with e.g. 2 clients. - Start Data Prepper and send Otel traces e.g. via the opentelemetry gen tool and an OTel collector inbetween.
- Observe the logs on DP and the client side.
Expected behavior
- consider returning a different status code which would allow retries.
Environment (please complete the following information):
- DP 2.11.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the data-prepper-plugins/otel-trace-source area linked in the issue and trace how connection-limit failures become gRPC responses. Reproduce with max_connection_count reduced to two and two clients, then verify that the response uses a retryable status such as UNAVAILABLE without changing other failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- backend-api-design, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100