redpanda-data / redpanda-data/connect

Add a weighted input broker

Open
#856 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement inputs ux
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

An interesting concept would be an input broker with one or more children, where each child is given a "weight" that determines how much priority in being consumed it has over other inputs.

For example, if we configured an input foo with a weight of 10, followed by an input bar with a weight of 0, then we would expect foo to be consumed exclusively unless it has no messages to provide, at which point we momentarily consume from bar until foo resumes again.

We generally handle back pressure gracefully in all inputs, and it's possible to write a broker using Go channels where we can dynamically select which inputs are being prioritized, so this should be possible to implement.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by locating the existing input and broker entry points and reviewing how input back pressure is handled. Done means a broker can prioritize child inputs by weight, consume exclusively from a higher-weight input while it has messages, and fall back to lower-weight inputs when it is empty.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.