kubeflow / kubeflow/notebooks

[TASK] Detect and recover from unhealthy Kind cluster state in make tilt-up

Open
#858 2 comments 0 reactions 0 assignees View on GitHub
area/ci area/v2 kind/plan-task
Dominant language
No language data
Stars
84
Forks
149
Avg merge
5d 15h
Merged PRs (30d)
29

Description

### Certification

- [x] I certify I am an Epic Owner for Kubeflow Notebooks 2.0 and expected to create planning-related issues.

### Description

The `make tilt-up` workflow does not handle the case where a Kind cluster named "tilt" exists but its API server is unreachable. This results in a confusing error during cert-manager installation rather than a clear diagnosis and recovery path.

### Observed Behavior

```bash
$ gmake tilt-up
Setting up Kind cluster...
Using KIND_EXPERIMENTAL_PROVIDER=podman
Kind cluster 'tilt' already exists
Switched to context "kind-tilt".
Kind cluster setup complete
Setting up cert-manager...
Installing cert-manager v1.12.13...
error: error validating "https://github.com/jetstack/cert-manager/releases/download/v1.12.13/cert-manager.yaml":
error validating data: failed to download openapi:
Get "https://127.0.0.1:54202/openapi/v2?timeout=32s": dial tcp 127.0.0.1:54202: connect: connection refused
```

The script reports "Kind cluster setup complete" but the cluster is not actually usable.

### Root Cause

In `developing/scripts/setup-kind.sh` (lines 19-26):

```bash
# Check if cluster exists
if ! kind get clusters 2>/dev/null | grep -q "^${CLUSTER_NAME}$"; then
echo "Creating Kind cluster '${CLUSTER_NAME}' with config from ${KIND_CONFIG}..."
kind create cluster --name "${CLUSTER_NAME}" --config "${KIND_CONFIG}" --wait 60s
echo "Kind cluster created successfully"
else
echo "Kind cluster '${CLUSTER_NAME}' already exists"
fi
```

The check `kind get clusters | grep tilt` only verifies the cluster **metadata** exists, not that:
- The Kind container(s) are running
- The API server is listening
- kubectl can successfully communicate with the cluster

### When This Happens

Common scenarios where the cluster exists but API server is down:
1. **Container runtime restarted**: Docker/Podman was restarted, Kind containers stopped
2. **Machine sleep/wake**: Laptop was suspended, containers didn't resume properly
3. **Resource exhaustion**: Kind container crashed due to OOM or disk pressure
4. **Port binding conflict**: Another process took the API server port after restart
5. **Partial cleanup**: Previous `kind delete` failed partway through

### Expected Behavior

The `make tilt-up` workflow should:
1. Detect when an existing cluster's API server is unreachable
2. Provide a clear error message explaining the situation
3. Offer a recovery path (e.g., delete and recreate the cluster)

### Key Files

| File | Role |
|------|------|
| `developing/Makefile` | Orchestrates `tilt-up` target |
| `developing/scripts/setup-kind.sh` | Checks/creates Kind cluster |
| `developing/scripts/setup-cert-manager.sh` | First kubectl command that fails |

### Concepts a Solution Should Consider

1. **Health check after existence check**: After confirming cluster exists, verify API server responds (e.g., `kubectl cluster-info` or `kubectl get nodes`)
2. **Timeout for health check**: Don't wait forever—give the cluster a reasonable window to respond
3. **Clear error messaging**: Tell the developer exactly what's wrong and what to do
4. **Automatic recovery option**: Optionally offer to delete and recreate the cluster
5. **Container runtime check**: For podman/docker, verify the kind control-plane container is actually running
6. **Idempotency**: Solution should handle repeated runs gracefully

### Example Recovery Flow

```bash
$ gmake tilt-up
Setting up Kind cluster...
Kind cluster 'tilt' exists but API server is not responding.

Possible causes:
- Docker/Podman containers may have stopped
- The cluster may need to be recreated

To fix, run:
kind delete cluster --name tilt
gmake tilt-up

Or to attempt container restart:
docker start tilt-control-plane # or podman equivalent
```

### Acceptance Criteria

- [ ] `make tilt-up` detects when Kind cluster exists but API server is unreachable
- [ ] A clear, actionable error message is displayed (not a kubectl validation error)
- [ ] The error message includes the specific recovery command(s) to run
- [ ] When the cluster is healthy, behavior is unchanged (no new delays or prompts)
- [ ] Works with both Docker and Podman container runtimes (`KIND_EXPERIMENTAL_PROVIDER`)

Contributor guide

Open the contributing guide

Research direction

Read developing/Makefile, developing/scripts/setup-kind.sh, and developing/scripts/setup-cert-manager.sh to trace the tilt-up workflow. Reproduce the case with an existing unhealthy Kind cluster using Docker or Podman, then verify the health-check behavior before cert-manager installation. Done means healthy clusters remain unchanged and unreachable clusters produce an actionable recovery message with the relevant kind delete and gmake tilt-up commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes, shell
Domain
cli, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.