opensearch-project / opensearch-project/data-prepper

[BUG] StatusCode.UNAVAILABLE grpc response when reaching armeria connection limit

Open
#6,546 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug

The OTel trace source is configured with a connection limit (max_connection_count) which defaults to 500.

In case this limit is reached Data Prepper logs sth. like:

[armeria-boss-http-*:21890] WARN  com.linecorp.armeria.server.ConnectionLimitingHandler - Dropped 10 connection(s) to limit the number of open connections to 500 

and returns a StatusCode.UNKNOWN:

<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "Received http2 header with status: 500"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:2, grpc_message:"Received http2 header with status: 500"}"

This status code is not classified as retryable failure.

Since clients are dropping the connection in this case, I am wondering if Data Prepper should respond with e.g. StatusCode.UNAVAILABLE instead so that this could be retried within the OTel client.

However, if a client consistently reaches the connection limit, retries could again fail of course.

To Reproduce
Steps to reproduce the behavior:

  1. Configure data prepper with the trace analytics setup.
  2. Reduce the max_connection_count to a lower number so that this is reproducible with e.g. 2 clients.
  3. Start Data Prepper and send Otel traces e.g. via the opentelemetry gen tool and an OTel collector inbetween.
  4. Observe the logs on DP and the client side.

Expected behavior

  • consider returning a different status code which would allow retries.

Environment (please complete the following information):

  • DP 2.11.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the data-prepper-plugins/otel-trace-source area linked in the issue and trace how connection-limit failures become gRPC responses. Reproduce with max_connection_count reduced to two and two clients, then verify that the response uses a retryable status such as UNAVAILABLE without changing other failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
backend-api-design, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.