redpanda-data / redpanda-data/redpanda
Configurable limit on concurrent metadata requests
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 12.6k
- Forks
- 792
- PR merge metrics
- No merged PRs in 30d
Description
Who is this for and what problem do they have today?
We're operating Redpanda with a large number of connected clients (64K in prod) and a large number of partitions (40K). With such a high partition count, topic metadata requests can be pretty heavy. When some kind of issue happens and partition leadership transfers happen, clients will start making more metadata requests to get up-to-date leadership information. The sudden influx of heavy metadata requests seems to be impacting Redpanda's ability to recover.
What are the success criteria?
A configuration option is provided which allows limiting the number of concurrent metadata requests.
Why is solving this problem impactful?
This seems to be impacting Redpanda's ability to recover when the cluster has a large number of partitions. This would allow Redpanda to shed load until it stabilizes.
Additional notes
JIRA Link: CORE-1206
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating the metadata request handling and configuration definitions, then determine where concurrent requests can be limited; done means a configurable limit reduces concurrent metadata work during recovery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100