devantler-tech / devantler-tech/ksail

fix(ci): EKS smoke test cluster create times out at 30 minutes with no progress output

Open
#6,369 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
165
Forks
12
Avg merge
5h 41m
Merged PRs (30d)
337

Description

> 🤖 Generated by the Agentic Engineer

## Evidence

First post-#6363 dispatch of `System Test - EKS` (run 30201658347, 2026-07-26, `main`, defaults: Flux / us-east-1):

```
##[error]The action '🧪 ksail cluster create' has timed out after 30 minutes.
```

The step is capped at `timeout-minutes: 30` while the job allows 120. Everything ahead of it passed — OIDC assumption, STS identity, permissions-boundary resolution, eksctl install, `ksail project init`. Teardown then reclaimed the cluster, so nothing leaked.

**The create step emitted no output at all across those 30 minutes.** The job log goes straight from the step's `##[group]Run` banner at 12:21:47Z to the timeout at 12:51:59Z — 23 log lines in the whole window, all of them belonging to the *teardown* that followed. There is no way to tell from this run whether the control plane was still provisioning, the managed nodegroup was mid-rollout, or the command was wedged.

## Problem / audience

This is the last blocker to the EKS smoke test ever completing end to end. The workflow has **never** had a genuine pass — the one "success" in its history (2026-07-19) skipped both the build and the smoke job, and the four real attempts before today failed on a cluster-naming defect fixed in #6308.

Two separate things are wrong, and the second is the one that will keep costing runs:

1. **The budget is too small.** An eksctl EKS create is control plane (~10 min) plus a managed nodegroup plus addons, and this run also had a GitOps bootstrap behind it. 30 minutes is not a generous allowance for that; it is roughly the expected duration.
2. **The step is silent, so a timeout teaches us nothing.** A 30-minute wall with no progress output means every failure looks identical — slow, hung, and stuck-retrying are indistinguishable, and each diagnosis costs another billable cluster.

## Smallest useful change

- Raise the create step's budget to something that fits an EKS create with headroom, staying inside the 120-minute job cap and leaving room for the `always()` teardown.
- Make the create observable: surface eksctl/provisioner progress as it happens rather than buffering it, so the log shows which phase it reached.

Fixing only the timeout would likely turn this red run into a slower red run with the same absent diagnosis.

## Acceptance criteria

- [ ] `ksail cluster create` for EKS completes within its step budget on a normal run, or fails with a log that names the phase it was in.
- [ ] The step streams progress while provisioning — a reader can tell control plane from nodegroup from addons.
- [ ] The teardown still runs and still verifies absence after a create timeout (unchanged from #6363).
- [ ] One end-to-end green run of `System Test - EKS` is recorded on the issue.

## Notes

Cost: each diagnosis attempt creates a real, billable cluster, so batch changes rather than dispatching per tweak. Teardown was verified working on this run — `ksail cluster delete` itself failed (#6368) and the `eksctl` fallback reclaimed everything.

This is the remaining acceptance criterion for devantler-tech/platform#2327, whose credential half is now proven: OIDC assumption and `sts get-caller-identity` both succeeded on this run. Rough size: S–M.

Contributor guide

Open the contributing guide

Research direction

Start with the `System Test - EKS` workflow and its `ksail cluster create` step; inspect how the 30-minute timeout and command output are configured. Run the EKS smoke test, then confirm provisioning phases stream in the log, the create fits the revised budget or reports its phase, and `always()` teardown still verifies cluster absence after timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions, kubernetes
Domain
ci-cd, cloud, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.