devantler-tech / devantler-tech/ksail

fix(cluster): a graceful API-server GOAWAY aborts a multi-step Kubernetes upgrade mid-sequence

Open
#6,848 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
165
Forks
12
Avg merge
5h 41m
Merged PRs (30d)
337

Description

> 🤖 Generated by the Agentic Engineer

## Evidence

`ksail cluster update` aborts a multi-step Kubernetes upgrade when the API server closes an HTTP/2
connection **gracefully** during its own restart. Observed on `ksail#6847`, job
[100488028950](https://github.com/devantler-tech/ksail/actions/runs/33703166551/job/100488028950)
(`🧪 System Test (Docker) (Talos, Docker, true, --gitops-engine ArgoCD ...)`), 2026-09-03T02:05:35Z:

```
✗ Kubernetes upgrade failed at step 3/3 (v1.36.4 → v1.37.0), cluster is running v1.36.4:
K8s upgrade to v1.37.0 failed: error applying manifests: failed to diff object
v1.ConfigMap/kube-system/kubeconfig-in-cluster, apply dry run failed:
ConfigMap/kube-system/kubeconfig-in-cluster dry-run failed:
Patch "https://10.5.0.2:6443/api/v1/.../kubeconfig-in-cluster?dryRun=All&fieldManager=talos&force=true":
http2: server sent GOAWAY and closed the connection; LastStreamID=89, ErrCode=NO_ERROR, debug=""
```

Steps 1/3 (`1.36.2 → 1.36.3`) and 2/3 (`1.36.3 → 1.36.4`) had both succeeded in the same invocation.

## Why this is a real defect and not just a flaky test

`ErrCode=NO_ERROR` is the decisive detail. GOAWAY with `NO_ERROR` is the HTTP/2 frame a server sends
to say *"I am shutting down cleanly, stop sending me new streams"* — it is the API server behaving
correctly while it restarts as part of the very upgrade being performed. The TLS connection was
established and healthy; nothing was wrong with the request.

The environment makes the race structural rather than unlucky: the run logs
`discovered controlplane nodes ["10.5.0.2"] discovered worker nodes []` — a **single-node control
plane**. With one API server there is no second replica to absorb the request, so every control-plane
restart opens a window in which any in-flight request gets exactly this GOAWAY. A single-node cluster
is ksail's most common shape (it is what the whole Docker/Talos system-test matrix uses, and what most
local users run), so this is reachable by real users upgrading a real cluster, not only by CI.

The failure is also maximally unhelpful: the upgrade aborts **mid-sequence**, leaving the cluster on
`v1.36.4` — neither the old nor the intended version — and the error surfaces as an opaque `http2:`
string rather than as "the API server restarted, retry".

## Likely origin — worth confirming before fixing

The surrounding output (`discovered controlplane nodes`, `checking for remaining...`, the
`fieldManager=talos` patch) indicates this is Talos's own `upgrade-k8s` path that ksail invokes, not
ksail's code. So the fix may belong upstream in `siderolabs/talos`. That is exactly what the first
step below is for — this issue should not assume ksail owns the retry until that is established.

## Acceptance criteria

1. Establish where the retry belongs: ksail's upgrade invocation, or Talos's `upgrade-k8s`. If
upstream, this issue becomes "carry a retry at the ksail boundary until upstream lands", and the
upstream report is prepared here rather than filed automatically.
2. A graceful `GOAWAY` / `NO_ERROR` (and the connection-reset class that accompanies an API-server
restart) is retried with backoff rather than aborting the upgrade step.
3. A retry that exhausts its budget reports the cause in plain terms — that the API server restarted —
instead of the raw `http2:` string.
4. Regression proof: a test that fails without the retry and passes with it, driving a GOAWAY at the
point the upgrade issues its dry-run patch. No `t.Skip`, and the transient is simulated rather than
waited for.

## Size

Small-to-medium, and dominated by criterion 1. If the retry belongs at the ksail boundary it is a
bounded wrapper around the upgrade step; if it is upstream, the deliverable here is the prepared
report plus a temporary boundary retry.

## Not doing now

I did not re-run the failed job to "fix" this — the transient is the evidence, and `ksail#6847` is a
rung-0 hotfix for red `main` whose own correctness is unaffected by this (the x/crypto bump cannot
cause a GOAWAY: the connection was already established and negotiated).

Contributor guide

Open the contributing guide

Research direction

Trace the `ksail cluster update` entry point and the Talos `upgrade-k8s` invocation to establish whether retries belong in ksail or upstream. Reproduce the dry-run patch failure described in the Docker/Talos system-test evidence, then add a bounded retry and regression test for graceful GOAWAY or connection-reset errors. Done means multi-step upgrades continue after a restart and exhausted retries explain that the API server restarted.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.