devantler-tech / devantler-tech/ksail
fix(cluster): a graceful API-server GOAWAY aborts a multi-step Kubernetes upgrade mid-sequence
- Dominant language
- Go
- Stars
- 165
- Forks
- 12
- Avg merge
- 5h 41m
- Merged PRs (30d)
- 337
Description
> 🤖 Generated by the Agentic Engineer
## Evidence
`ksail cluster update` aborts a multi-step Kubernetes upgrade when the API server closes an HTTP/2
connection **gracefully** during its own restart. Observed on `ksail#6847`, job
[100488028950](https://github.com/devantler-tech/ksail/actions/runs/33703166551/job/100488028950)
(`🧪 System Test (Docker) (Talos, Docker, true, --gitops-engine ArgoCD ...)`), 2026-09-03T02:05:35Z:
```
✗ Kubernetes upgrade failed at step 3/3 (v1.36.4 → v1.37.0), cluster is running v1.36.4:
K8s upgrade to v1.37.0 failed: error applying manifests: failed to diff object
v1.ConfigMap/kube-system/kubeconfig-in-cluster, apply dry run failed:
ConfigMap/kube-system/kubeconfig-in-cluster dry-run failed:
Patch "https://10.5.0.2:6443/api/v1/.../kubeconfig-in-cluster?dryRun=All&fieldManager=talos&force=true":
http2: server sent GOAWAY and closed the connection; LastStreamID=89, ErrCode=NO_ERROR, debug=""
```
Steps 1/3 (`1.36.2 → 1.36.3`) and 2/3 (`1.36.3 → 1.36.4`) had both succeeded in the same invocation.
## Why this is a real defect and not just a flaky test
`ErrCode=NO_ERROR` is the decisive detail. GOAWAY with `NO_ERROR` is the HTTP/2 frame a server sends
to say *"I am shutting down cleanly, stop sending me new streams"* — it is the API server behaving
correctly while it restarts as part of the very upgrade being performed. The TLS connection was
established and healthy; nothing was wrong with the request.
The environment makes the race structural rather than unlucky: the run logs
`discovered controlplane nodes ["10.5.0.2"] discovered worker nodes []` — a **single-node control
plane**. With one API server there is no second replica to absorb the request, so every control-plane
restart opens a window in which any in-flight request gets exactly this GOAWAY. A single-node cluster
is ksail's most common shape (it is what the whole Docker/Talos system-test matrix uses, and what most
local users run), so this is reachable by real users upgrading a real cluster, not only by CI.
The failure is also maximally unhelpful: the upgrade aborts **mid-sequence**, leaving the cluster on
`v1.36.4` — neither the old nor the intended version — and the error surfaces as an opaque `http2:`
string rather than as "the API server restarted, retry".
## Likely origin — worth confirming before fixing
The surrounding output (`discovered controlplane nodes`, `checking for remaining...`, the
`fieldManager=talos` patch) indicates this is Talos's own `upgrade-k8s` path that ksail invokes, not
ksail's code. So the fix may belong upstream in `siderolabs/talos`. That is exactly what the first
step below is for — this issue should not assume ksail owns the retry until that is established.
## Acceptance criteria
1. Establish where the retry belongs: ksail's upgrade invocation, or Talos's `upgrade-k8s`. If
upstream, this issue becomes "carry a retry at the ksail boundary until upstream lands", and the
upstream report is prepared here rather than filed automatically.
2. A graceful `GOAWAY` / `NO_ERROR` (and the connection-reset class that accompanies an API-server
restart) is retried with backoff rather than aborting the upgrade step.
3. A retry that exhausts its budget reports the cause in plain terms — that the API server restarted —
instead of the raw `http2:` string.
4. Regression proof: a test that fails without the retry and passes with it, driving a GOAWAY at the
point the upgrade issues its dry-run patch. No `t.Skip`, and the transient is simulated rather than
waited for.
## Size
Small-to-medium, and dominated by criterion 1. If the retry belongs at the ksail boundary it is a
bounded wrapper around the upgrade step; if it is upstream, the deliverable here is the prepared
report plus a temporary boundary retry.
## Not doing now
I did not re-run the failed job to "fix" this — the transient is the evidence, and `ksail#6847` is a
rung-0 hotfix for red `main` whose own correctness is unaffected by this (the x/crypto bump cannot
cause a GOAWAY: the connection was already established and negotiated).
Contributor guide
Research direction
Trace the `ksail cluster update` entry point and the Talos `upgrade-k8s` invocation to establish whether retries belong in ksail or upstream. Reproduce the dry-run patch failure described in the Docker/Talos system-test evidence, then add a bounded retry and regression test for graceful GOAWAY or connection-reset errors. Done means multi-step upgrades continue after a restart and exhausted retries explain that the API server restarted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100