devantler-tech / devantler-tech/ksail

ci: restore the weekly cloud-provider system-test schedule

Open
#6,361 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
165
Forks
12
Avg merge
5h 41m
Merged PRs (30d)
337

Description

> 🤖 Generated by the Agentic Engineer

## Evidence

The weekly cloud-provider system tests **used to run and were deliberately paused**, so this is a restoration rather than a new capability.

- Hetzner and Omni ran on `0 3 * * 0` and `0 4 * * 0` (staggered Sunday cron). The last scheduled runs were 2026-06-21, 06-28 and 07-05 — all red.
- `e8d6566b` (2026-07-09) removed both triggers. The reason is recorded in each workflow: the failures were credential/infrastructure noise, not product signal.
- **Hetzner** — `HCLOUD_TOKEN` invalid/expired since ~2026-04-26, so every run died at the cluster-existence check before provisioning anything (tracked in #4972).
- **Omni** — zero machines registered in the Omni instance since ~2026-04-26, so every run died at machine auto-discovery (tracked in #4973).
- **EKS** has never been scheduled, by choice: the workflow notes the smoke test creates billable AWS resources. Its recent dispatch runs (07-19, 07-21) are also red.
- **#4972 and #4973 are both closed, but the schedules are still paused.** Nothing currently tracks re-enabling them, which is the gap this issue fills.

Worth noting the scheduled path is **already implemented**: the Omni workflow expands its matrix to four configurations when `github.event_name == schedule` for broad coverage. The missing piece is the trigger and the infrastructure behind it, not the test.

## Problem

Since 2026-07-05 there has been **no automated regression signal on any cloud provider**. Changes to provider create/reconcile/delete paths land with only local and unit coverage. A regression against a real provider now surfaces either when someone remembers to dispatch a run manually, or when a user hits it — which is exactly the silent-regression risk this schedule existed to prevent.

The prior failure mode also matters: the tests sat red for six consecutive weeks before anyone acted. Re-enabling a cron without fixing that feedback gap would recreate the noise that justified pausing it.

## What this needs

Three independent blockers, one per provider:

1. **Hetzner** — rotate `HCLOUD_TOKEN`, confirm a dispatch run goes green, then restore `0 3 * * 0`.
2. **Omni** — register a machine in the Omni instance, confirm a dispatch run goes green, then restore `0 4 * * 0`.
3. **EKS** — decide the cost posture rather than defaulting: schedule it and accept the spend, keep it dispatch-only, or schedule a reduced/cheaper matrix. This is a judgement call, not a config edit.

## Acceptance criteria

- Each provider either has a weekly schedule **and a green scheduled run on record**, or a recorded decision not to schedule it with the reason kept in the workflow.
- A red scheduled run becomes visible promptly instead of accumulating silently — the six-week silent-red pattern must not repeat.
- Cleanup is verified on failure paths, not just success, so a failed run cannot leak billable resources.
- The per-provider cost posture is written down where the next person will find it.

## Guardrails

- Hetzner and EKS provision billable infrastructure; Omni consumes registered machines.
- The portfolio rule that real clusters are not spun up more than once a day applies — weekly per provider, staggered by an hour, already respects it.

Rough size: medium. The cron restoration itself is trivial; the real work is unblocking the two credential/infrastructure dependencies and making the EKS cost decision. Likely worth decomposing into one child per provider.

Contributor guide

Open the contributing guide

Research direction

Start by reading the Hetzner, Omni, and EKS workflow definitions, including their existing schedule notes and dispatch paths. Verify the provider credentials or registered infrastructure, run each provider manually before changing schedules, and consider the cost and cleanup requirements; done means a green scheduled run or a documented decision not to schedule each provider.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions, kubernetes
Domain
ci-cd, cloud, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.