devantler-tech / devantler-tech/ksail
Autoscaler-provisioned Hetzner nodes lack ksail.io/autoscaled and keep the Longhorn disk label
- Dominant language
- Go
- Stars
- 165
- Forks
- 12
- Avg merge
- 5h 41m
- Merged PRs (30d)
- 337
Description
> 🤖 Generated by the Agentic Engineer
## Evidence
Measured read-only against the devantler-tech production cluster (Talos + Hetzner, KSail-managed Cluster Autoscaler), 2026-09-13 ~04:30Z. KSail v7.184.0 is the current release.
The autoscaler nodes do **not** carry the marker that `GenerateAutoscalerWorkerConfig` is documented to stamp, and they **do** carry the Longhorn label it is documented to strip:
| node | created | `ksail.io/autoscaled` | `node.longhorn.io/create-default-disk` |
|---|---|---|---|
| `autoscale-cx43-59ee3c84…` | 2026-08-26 | **absent** | **`true`** |
| `autoscale-cx43-55e35623…` | 2026-09-08 | **absent** | **`true`** |
| `prod-worker-{1,2,3}` (static) | 2026-06-16 | absent | `true` |
Apart from the hostname, the label sets of the autoscaler nodes and the static workers are identical. Both autoscaler nodes were created months after ksail#5113 (label stamping) and ksail#5219 (template regeneration) closed.
The Cluster Autoscaler Deployment reads `HCLOUD_CLUSTER_CONFIG` from the `kube-system/cluster-autoscaler-config` Secret, and runs pools `autoscale-cx43` and `autoscale-cx53` with image `cluster-autoscaler:v1.35.0`. I did not read the Secret's contents; the observation is limited to the node labels.
## Problem and impact
A node carrying the full worker label set and no marker did not boot from the stripped autoscaler worker config. So either the template stored in that Secret is not what `GenerateAutoscalerWorkerConfig` produces, or the marker and stripping are lost between the template and the booted node.
Every consumer of `ksail.io/autoscaled` is therefore silently inert on this cluster:
- The platform's soft FinOps bias toward the static workers keys on `DoesNotExist`, so it admits every node.
- The platform's hard pin that keeps OpenBao's single-attach hcloud volumes off autoscaler nodes admits every node too. At measurement time, 9 hcloud `VolumeAttachment`s were on autoscaler nodes. A volume stranded on a reclaimed node previously wedged production delivery for ~9h (devantler-tech/platform#2363).
The platform is adding a positive, name-keyed label to its static workers so its guards stop depending on this marker (devantler-tech/platform#3287). The marker itself is still broken for every other KSail user.
## Expected behaviour
A node the KSail-managed Cluster Autoscaler provisions boots from the stripped autoscaler worker config: it carries `ksail.io/autoscaled=true` and does not carry `node.longhorn.io/create-default-disk`.
## Acceptance criteria
- [ ] Root cause identified: stale Secret template, regeneration not triggered by the deploy path used, or loss between template and boot.
- [ ] A newly provisioned autoscaler node carries `ksail.io/autoscaled=true` and lacks the Longhorn default-disk label. Verify this on a real node, not by inspecting the generated config.
- [ ] A regression test pins that the template the autoscaler actually consumes contains the marker.
## Rough size
S–M: diagnosis first.
Contributor guide
Research direction
Start at GenerateAutoscalerWorkerConfig and trace the deploy path that populates the kube-system/cluster-autoscaler-config Secret. Compare the consumed template with the expected marker and Longhorn-label behavior, then verify a newly provisioned autoscaler node and add a regression test for the actual autoscaler template.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100