tilt-dev / tilt-dev/tilt

cluster liveness check fails

Open
#5,976 13 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement needs repro case
Dominant language
Go
Stars
10.1k
Forks
413
Avg merge
1d 10h
Merged PRs (30d)
6

Description

Current Behavior

ahy in the slack channel reports that when they connect tilt to their remote cluster, it fails with:

Cluster status error: cluster did not pass liveness check

If they try to run the liveness check manually, they get

$ kubectl get --raw='/livez?verbose'
Error from server (NotFound): the server could not find the requested resource

It appears that livez was added in Kubernetes 1.16 and is not supported on their Rancher distro.

The confirm the /healthz check works though

Possible Solutions

Maybe we should only use /healthz? not sure what the additional benefit of using /livez is.

Alternatively, if we get a 404 from /livez, we could ignore it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with kubectl get --raw='/livez?verbose' against the affected Rancher-based cluster, then compare it with the working /healthz check. Review how Tilt performs the cluster liveness check and determine what behavior is expected when /livez returns 404. Done means the remote cluster no longer fails liveness checks solely because /livez is unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.