liquidmetal-dev / liquidmetal-dev/cluster-api-provider-microvm

Readiness probe gets stuck in endless restart loop

Open
#231 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug
Dominant language
Go
Stars
157
Forks
21
PR merge metrics
No merged PRs in 30d

Description

What happened:
[A clear and concise description of what the bug is.]

The CAPMVM controller's readiness probe gets stuck in an endless loop of restarts:

 Warning  Unhealthy    9h (x2 over 13h)      kubelet  Liveness probe failed: Get "http://10.244.0.10:9440/healthz": dial tcp 10.244.0.10:9440: connect: connection refused
 Warning  Unhealthy    9h (x8 over 15h)      kubelet  Readiness probe failed: Get "http://10.244.0.10:9440/readyz": dial tcp 10.244.0.10:9440: connect: connection refused

In the logs above^, the liveness probe (which occurs right before the readiness probe) passes much faster and does not get stuck in a loop of restarts, probably because it is configured to have more time:

# The initial delay for the liveness probe is intentionally large to
# avoid an endless kill & restart cycle if in the event that the initial
# bootstrapping takes longer than expected.
initialDelaySeconds: 120

What did you expect to happen:
I expected the readiness probe to not get stuck as much / wait for longer, similarly to the liveness probe :)

How to reproduce it:
I am using docker.io/niki2401/flintlock-kernel:5.10.77 as a kernel image and mounting it as an additional volume as well.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

  • capmvm version:v0.7.0
  • OS (e.g. from /etc/os-release):

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named in the issue. Start by locating the CAPMVM controller deployment's readiness and liveness probe configuration, then reproduce with the stated kernel image and compare their startup timing. Done means the readiness probe no longer causes an endless restart loop while bootstrapping.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.