devantler-tech / devantler-tech/ksail
EKS smoke test: the create step's 30-minute timeout is shorter than EKS provisioning takes
- Dominant language
- Go
- Stars
- 165
- Forks
- 12
- Avg merge
- 5h 51m
- Merged PRs (30d)
- 347
Description
> 🤖 Generated by the Agentic Engineer
### Evidence
`.github/workflows/system-test-eks.yaml` gives the `🧪 ksail cluster create` step
`timeout-minutes: 30`, while the job it sits in has `timeout-minutes: 210`. The step budget is the
binding one, and it is too small for real EKS provisioning.
Every dispatch of this workflow to date:
| run | date | outcome |
|---|---|---|
| 30201658347 | 2026-07-26 (`main`) | create ran **30m12s** (12:21:47→12:51:59) and was killed by the step timeout |
| 29822971789 | 2026-07-21 | create failed in **13s** — early error |
| 29694390718 | 2026-07-19 | create failed in **1s** — early error |
| 29694161648 | 2026-07-19 | create failed in **2s** — early error |
| 29693571225 | 2026-07-19 | failed at `🔧 Initialize EKS project` in **<1s** |
| 29669336152 | 2026-07-19 | run reports `success` but the smoke **job was skipped** — only the credentials check ran |
Note the last row: that run is *not* evidence of a passing smoke test, and it is easy to misread as
one because the run-level conclusion is `success`.
The shape of the trajectory matters. The four sub-15-second failures are early configuration errors
that have since been fixed; the 2026-07-26 run is **the first dispatch ever to reach real
provisioning**, and it died purely on the clock. At the moment it was killed, `eksctl` was still
polling the `eksctl-st-eks-30201658347-1-nodegroup-default` CloudFormation stack — i.e. it was
progressing normally, not stuck.
Teardown then worked correctly (`all cluster resources were deleted`, `No cluster … remains in
us-east-1`), so this is not leaving paid AWS resources behind — worth stating, since a timeout during
provisioning is exactly the case where orphaned infrastructure would be expensive.
### Impact
This is the current frontier blocker for #4328's "CI system test or smoke test covers the EKS path"
criterion, and it makes that criterion unreachable by construction: an EKS control plane plus a
managed nodegroup routinely takes longer than 30 minutes, so the step cannot pass no matter how
correct the code under test is.
It also fails in a **misleading** way. The symptom is a red `ksail cluster create`, which reads as
"KSail cannot create EKS clusters" — a product defect — when the actual cause is a CI budget. That
misattribution is the expensive part: it points investigation at the provisioner rather than at the
workflow.
### Expected behaviour
The create step is given a budget derived from how long EKS provisioning actually takes, within the
job's existing 210-minute envelope.
### Acceptance criteria
- [ ] The `🧪 ksail cluster create` step's `timeout-minutes` is large enough to cover control-plane
plus nodegroup creation with margin, and stays comfortably inside the job's 210-minute budget
so the remaining steps and teardown still fit.
- [ ] The value is justified from observed durations rather than picked round — and the reasoning
recorded, so the next person does not re-derive it.
- [ ] Teardown remains guaranteed on timeout, so a slow run still cannot orphan paid AWS resources.
This already holds today and must not regress.
- [ ] Verified by an actual dispatch reaching a green smoke run. Note this needs one real EKS
cluster, which is a genuine cost and is capped at one live cluster per day portfolio-wide, so
the verifying dispatch should be deliberate rather than casual.
Part of #4328.
Rough size: small change, but it cannot be called done until a live dispatch proves it — that is the
expensive half.
Contributor guide
Research direction
Read .github/workflows/system-test-eks.yaml, focusing on the 🧪 ksail cluster create step and the surrounding job timeout. Use the observed provisioning duration and the 210-minute job budget to justify a larger step timeout while preserving the existing teardown behavior. Verify the result with one deliberate EKS workflow dispatch that reaches a green smoke run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, github-actions, kubernetes
- Domain
- ci-cd, cloud, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 67/100