devantler-tech / devantler-tech/ksail
ci(eks): exercise in-place node-group scaling in the EKS smoke test
- Dominant language
- Go
- Stars
- 165
- Forks
- 12
- Avg merge
- 5h 51m
- Merged PRs (30d)
- 347
Description
> 🤖 Generated by the Agentic Engineer
## Evidence
#6196 has been open since 2026-07-16 and #6267 has been a parked draft since 2026-07-19. Both are blocked on the *same* acceptance criterion, worded as a manual act:
> The in-place scaling path is exercised against a real EKS cluster (scale a managed node group up and down via `cluster update`; verify no recreation and correct converged sizes).
Nothing in CI exercises that path today, and no one has run it by hand in ten days. Meanwhile `System Test - EKS` already creates a real EKS cluster with a managed node group, authenticates via OIDC, and — since #6363 — tears the cluster down behind a post-condition-verified safety net. The one capability the criterion needs already exists in CI; it simply is not exercised.
## Problem / audience
EKS operators: `spec.cluster.eks.experimentalInPlaceUpdates` stays default-off, so the standard update flow remains delete-and-recreate. The flag also accrues flag debt — the portfolio flag-lifecycle rule requires a release flag to be short-lived, and this one cannot be retired while its validation depends on someone manually provisioning a cluster.
## Hypothesis
Folding a scale-up / scale-down assertion into the existing smoke test converts a one-shot manual observation into a repeatable CI signal, satisfying #6196's first criterion on every run rather than once. That unblocks the graduation (#6196) and the parked PR (#6267) together, at no additional cluster cost — the cluster is already created and already paid for by the run.
## Success signal
`System Test - EKS` fails when in-place scaling regresses (node group recreated, or converged size wrong) and passes when it works. Baseline today: the path has **never** been exercised in CI. Guardrail: no measurable increase in cluster lifetime beyond the scaling operations themselves, and no change to the teardown path.
## Smallest useful change
Inside the existing `🧪 EKS Smoke Test` job, after the cluster is up and before teardown:
1. Record the managed node group's identity and current size (`eksctl get nodegroup` — capture the node group's creation timestamp / ARN and `desiredCapacity`).
2. Set `spec.cluster.eks.experimentalInPlaceUpdates: true` and change the node group's desired size in `eks.yaml`, using the repo's `yq` idiom (see #6313 — do not add another inline Ruby edit).
3. Run `ksail cluster update`.
4. Assert: the node group's creation timestamp / ARN is **unchanged** (proving no recreation) and the converged size matches the requested size.
5. Scale back down and assert the same two properties again.
Teardown is unchanged — it already runs under `if: always()`.
## Acceptance criteria
- [ ] The smoke test scales a managed node group up and down through `ksail cluster update` with `experimentalInPlaceUpdates` enabled.
- [ ] The no-recreation property is asserted against a stable node-group identity, not inferred from the command's exit status.
- [ ] Converged size is asserted after each direction, not just requested.
- [ ] A regression in the in-place path turns the job red (RED-proved before merge — e.g. against the flag disabled, which must recreate and therefore fail the identity assertion).
- [ ] Teardown behaviour and the existing `delete-eks-smoke-cluster.sh` post-condition check are untouched.
## Notes
- Depends on `System Test - EKS` being green end-to-end. As of filing, the first post-#6363 dispatch is in flight (run 30201658347); the last completed real attempt failed on 2026-07-21 for a cause since fixed (#6308, `eb9675915a`, on `main`). If that run reveals further breakage in the chain, fix that first — this issue sits on top of a working smoke test.
- The 2026-07-19 "success" in this workflow's history is a false green: preflight soft-skipped and both downstream jobs were `skipped`. That soft-skip path no longer exists.
Part of #6196. Unblocks #6267. Rough size: S–M.
Contributor guide
Research direction
Start with the existing System Test - EKS job, eks.yaml, and the yq idiom referenced by #6313; also review delete-eks-smoke-cluster.sh and wait for the workflow to be green. Done means the smoke test scales the managed node group up and down through ksail cluster update, verifies stable identity and converged size after each direction, and leaves teardown unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, github-actions, kubernetes, yaml
- Domain
- ci-cd, cloud, devops, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100