redpanda-data / redpanda-data/connect
Add a weighted input broker
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
An interesting concept would be an input broker with one or more children, where each child is given a "weight" that determines how much priority in being consumed it has over other inputs.
For example, if we configured an input foo with a weight of 10, followed by an input bar with a weight of 0, then we would expect foo to be consumed exclusively unless it has no messages to provide, at which point we momentarily consume from bar until foo resumes again.
We generally handle back pressure gracefully in all inputs, and it's possible to write a broker using Go channels where we can dynamically select which inputs are being prioritized, so this should be possible to implement.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by locating the existing input and broker entry points and reviewing how input back pressure is handled. Done means a broker can prioritize child inputs by weight, consume exclusively from a higher-weight input while it has messages, and fall back to lower-weight inputs when it is empty.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100