percona / percona/percona-server-mongodb-operator
Reconciliation calls kube-apiserver 1000/sec with high concurrency and many databases
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 456
- Forks
- 198
- Avg merge
- 5d 19h
- Merged PRs (30d)
- 10
Description
Report
We have 100 PSMDB defined. The operator is calling kube-apiserver over 1000x/sec with a MAX_CONCURRENT_RECONCILES of 60. Reducing to 2 lowers it to merely dozens of calls per second.
More about the problem
We have EKS audit logging enabled and the increased concurrency is causing a noticeable increase to our cloudwatch bill.
Steps to reproduce
- Be on operator 1.21.1
- Turn up MAX_CONCURRENT_RECONCILES
- Define many clusters
Versions
- Kubernetes v1.32
- Operator v1.21.1
- Database 6-8 (various versions)
Anything else?
I reported it as a bug because I'm not sure this level of activity is intended. But maybe this is actually a feature request for a parameter to slow down the reconciliation loop.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the operator's MAX_CONCURRENT_RECONCILES setting and reproduce the reconciliation traffic using many PSMDB clusters on operator 1.21.1. Determine whether the observed kube-apiserver call rate is expected, then establish whether the fix should reduce unnecessary calls or expose a way to limit reconciliation; done means the behavior is verified and the expected call rate or control is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, mongodb
- Domain
- databases, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100