redpanda-data / redpanda-data/redpanda

redpanda: add a configuration to support out-of-order responses to client requests

Open
#4,898 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/enhance mt
Dominant language
C++
Stars
12.6k
Forks
792
PR merge metrics
No merged PRs in 30d

Description

Apparently, the original implementation of Redpanda allowed responses to be returned out of order: nothing within seastar enforces ordering. This implementation was changed in support of the Kafka protocol of ensuring all requests are replied to in order, per the correlation ID.

We should add configuration to opt in to returning responses out of order. There are a few potential ways we could do this:

  1. A broker level configuration, such that all responses are delivered when they're ready. This seems like an overly broad solution that can be problematic if you want to use literally any client that does not support out of order responses (i.e., all clients currently)
  2. A topic level configuration. This is an acceptable middle ground, where we can expect that any client that talks to a specific topic support out-of-order responses. However, this is not as generic and client-supporting as...
  3. A client<=>broker configuration that is negotiated when the client dials. We can abuse Kafka's flexible requests and tags to allow the client to specify a Redpanda specific tag in an ApiVersions request header. A well written client sends ApiVersions as the first request on a connection. If we add a custom tag, OutOfOrderResponses, the client can opt into out of order responses on an individual connection. This is the most generic option and allows non-out-of-order-response clients to talk to the same topics that out-of-order-response clients are talking to.

The benefit of out of order responses is that this allows a client to open many fewer connections to the broker, and allows each connection to have a higher window size. Basically, this is moving closer to an http2 model rather than an http1 model (although http2 does have head of line blocking: a client can be a bit smarter and use a small pool of connections, or something).

JIRA Link: CORE-921

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Kafka's flexible ApiVersions request and the seastar response-ordering behavior described in the issue. Resolve the configuration scope, then implement an opt-in mechanism that lets compatible clients receive responses out of order while preserving ordered responses for other connections.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka
Domain
api, distributed-systems, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.