devantler-tech / devantler-tech/ksail

An out-of-band-deleted EKS cluster can no longer be cleared through the local API

Closed
#6,435 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
165
Forks
12
Avg merge
5h 51m
Merged PRs (30d)
347

Description

> 🤖 Generated by the Agentic Engineer

## Evidence

#6434 switches on the immutable-ownership guard for local API EKS mutations (part of #6203). One
consequence, surfaced while self-reviewing that PR and stated on it rather than left to be
discovered:

`runDelete` deliberately treats `clustererr.ErrClusterNotFound` as success and proceeds to clean up
local state. Its own comment records why:

> Deleting must be idempotent. ErrClusterNotFound means there is nothing to delete — a failed create
> that produced no cluster, a cluster removed out-of-band, or a race where it vanished between
> enumeration and Delete. […] otherwise the job stays pinned Failed and the UI row can never be
> dismissed (forcing an app restart or a fallback to `ksail cluster delete --name`).

The guard now runs **before** that, and verification requires `DescribeCluster` to succeed. For an
EKS cluster that has genuinely vanished from AWS — deleted in the console, or removed by another
operator — the guard fails and the delete never reaches its idempotent cleanup.

The standalone CLI path (`ensureAWSClusterManaged`) has the same property, so the two surfaces are
now consistent; this is not a regression *relative to the CLI*. It is a real reduction in what the
web UI can recover from, and it makes one existing message inaccurate: the create-refusal in
`distconfig.go` promises `ksail cluster delete --name ` "succeeds even when the cluster is already
gone", which is now weaker than its wording for EKS.

## Affected audience and impact

An operator whose EKS cluster was removed out-of-band is left with a Failed row they cannot dismiss
and local state (`spec.json`, `eks.yaml`, nodegroup snapshot) they cannot clear through either
surface. No data is at risk and nothing is silently wrong — the failure is loud — but the recovery
story is a dead end, and the message that names the way out no longer describes one.

## Expected behaviour

A genuinely-absent cluster should be distinguishable from an unverifiable one, and only the first
should permit local-state cleanup. Fail-closed must remain the default: "cannot describe the
cluster" is otherwise indistinguishable from a credential pointed at the wrong account, an expired
session, or a transient AWS error, and that ambiguity is exactly what #6203 asks the backend to
refuse on.

## Acceptance criteria

- [ ] The identity client distinguishes **cluster-does-not-exist** (AWS `ResourceNotFoundException`,
and the existing `eks.ErrClusterNotFound` empty-payload case) from every other describe failure,
as a typed result rather than string matching.
- [ ] Only that distinguished absence permits local-state cleanup; a credential, permission,
throttling, or network failure still refuses. A test covers each arm, including the
wrong-account case, so the carve-out cannot widen into "any describe error means gone".
- [ ] The absence path still refuses anything that would touch AWS — it cleans local state only.
- [ ] The `distconfig.go` create-refusal message is re-checked against whatever recovery path is
chosen, so it does not promise something that no longer works.
- [ ] Decide explicitly whether the same treatment applies to start/stop, or only to delete —
scaling a nodegroup on an absent cluster has no cleanup semantics to preserve.

Follow-up to #6434. Related: #6203 (criterion 3), #6202.

Contributor guide

Open the contributing guide

Research direction

Start with runDelete, the identity client’s DescribeCluster handling, and distconfig.go, then inspect the existing EKS error and cleanup tests. Define a typed not-found result, preserve refusal for other describe failures including wrong-account access, and verify that only the absent-cluster path cleans local state without touching AWS; also resolve the start/stop scope and re-check the recovery message.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
api, cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.