SovereignCloudStack / SovereignCloudStack/cluster-stacks
Leaking kubeadmconfigtemplates, openstackmachinetemplates ...
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
/kind bug
What steps did you take and what happened:
A management cluster (kind) running in an SCS-2V-4 VM for 3 months (mostly idle) became unusable.
After some debugging, it was found that the kube-apiserver's memory usage had exploded to > 2GiB RSS.
This caused the machine to aggressively discard memory (kswapd0) just to hit major page faults resulting in the memory to be paged back in. System load > 50 (on a 2vCPU server), >>10k major page faults/s and >500MB/s reading from disk.
What did you expect to happen:
4GiB should be sufficient RAM for a not too busy management host.
Anything else you would like to add:
I was assuming that the CSO/CSPO are causing the kube-apiserver memory usage by storing too many objects.
I thus far found kubeadmconfigtemplates and clusterclasses to exist in excessive numbers.
Environment:
- kind v0.20.0 go1.20.4 linux/amd64
- Ubuntu 22.04 VM on an SCS-2V-4 flavor (2vCPU, 4GiB RAM, x86-64)
- CSO/CSPO as of 93d ago (let me know how I can report this better)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by investigating the kube-apiserver memory growth and the excessive kubeadmconfigtemplates and clusterclasses reported in the management cluster. Examine whether CSO/CSPO are creating or retaining those objects over time. Done means identifying the source of the leak and preventing the object accumulation, with sustained memory usage remaining viable on the stated 4GiB environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, linux, ubuntu
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100