SovereignCloudStack / SovereignCloudStack/cluster-stacks

Leaking kubeadmconfigtemplates, openstackmachinetemplates ...

Open
#105 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13
Forks
8
PR merge metrics
No merged PRs in 30d

Description

/kind bug

What steps did you take and what happened:
A management cluster (kind) running in an SCS-2V-4 VM for 3 months (mostly idle) became unusable.
After some debugging, it was found that the kube-apiserver's memory usage had exploded to > 2GiB RSS.
This caused the machine to aggressively discard memory (kswapd0) just to hit major page faults resulting in the memory to be paged back in. System load > 50 (on a 2vCPU server), >>10k major page faults/s and >500MB/s reading from disk.

What did you expect to happen:
4GiB should be sufficient RAM for a not too busy management host.

Anything else you would like to add:
I was assuming that the CSO/CSPO are causing the kube-apiserver memory usage by storing too many objects.
I thus far found kubeadmconfigtemplates and clusterclasses to exist in excessive numbers.

Environment:

  • kind v0.20.0 go1.20.4 linux/amd64
  • Ubuntu 22.04 VM on an SCS-2V-4 flavor (2vCPU, 4GiB RAM, x86-64)
  • CSO/CSPO as of 93d ago (let me know how I can report this better)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by investigating the kube-apiserver memory growth and the excessive kubeadmconfigtemplates and clusterclasses reported in the management cluster. Examine whether CSO/CSPO are creating or retaining those objects over time. Done means identifying the source of the leak and preventing the object accumulation, with sustained memory usage remaining viable on the stated 4GiB environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, linux, ubuntu
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.