devantler-tech / devantler-tech/ksail

Autoscaler-provisioned Hetzner nodes lack ksail.io/autoscaled and keep the Longhorn disk label

Open
#7,013 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
165
Forks
12
Avg merge
5h 41m
Merged PRs (30d)
337

Description

> 🤖 Generated by the Agentic Engineer

## Evidence

Measured read-only against the devantler-tech production cluster (Talos + Hetzner, KSail-managed Cluster Autoscaler), 2026-09-13 ~04:30Z. KSail v7.184.0 is the current release.

The autoscaler nodes do **not** carry the marker that `GenerateAutoscalerWorkerConfig` is documented to stamp, and they **do** carry the Longhorn label it is documented to strip:

| node | created | `ksail.io/autoscaled` | `node.longhorn.io/create-default-disk` |
|---|---|---|---|
| `autoscale-cx43-59ee3c84…` | 2026-08-26 | **absent** | **`true`** |
| `autoscale-cx43-55e35623…` | 2026-09-08 | **absent** | **`true`** |
| `prod-worker-{1,2,3}` (static) | 2026-06-16 | absent | `true` |

Apart from the hostname, the label sets of the autoscaler nodes and the static workers are identical. Both autoscaler nodes were created months after ksail#5113 (label stamping) and ksail#5219 (template regeneration) closed.

The Cluster Autoscaler Deployment reads `HCLOUD_CLUSTER_CONFIG` from the `kube-system/cluster-autoscaler-config` Secret, and runs pools `autoscale-cx43` and `autoscale-cx53` with image `cluster-autoscaler:v1.35.0`. I did not read the Secret's contents; the observation is limited to the node labels.

## Problem and impact

A node carrying the full worker label set and no marker did not boot from the stripped autoscaler worker config. So either the template stored in that Secret is not what `GenerateAutoscalerWorkerConfig` produces, or the marker and stripping are lost between the template and the booted node.

Every consumer of `ksail.io/autoscaled` is therefore silently inert on this cluster:

- The platform's soft FinOps bias toward the static workers keys on `DoesNotExist`, so it admits every node.
- The platform's hard pin that keeps OpenBao's single-attach hcloud volumes off autoscaler nodes admits every node too. At measurement time, 9 hcloud `VolumeAttachment`s were on autoscaler nodes. A volume stranded on a reclaimed node previously wedged production delivery for ~9h (devantler-tech/platform#2363).

The platform is adding a positive, name-keyed label to its static workers so its guards stop depending on this marker (devantler-tech/platform#3287). The marker itself is still broken for every other KSail user.

## Expected behaviour

A node the KSail-managed Cluster Autoscaler provisions boots from the stripped autoscaler worker config: it carries `ksail.io/autoscaled=true` and does not carry `node.longhorn.io/create-default-disk`.

## Acceptance criteria

- [ ] Root cause identified: stale Secret template, regeneration not triggered by the deploy path used, or loss between template and boot.
- [ ] A newly provisioned autoscaler node carries `ksail.io/autoscaled=true` and lacks the Longhorn default-disk label. Verify this on a real node, not by inspecting the generated config.
- [ ] A regression test pins that the template the autoscaler actually consumes contains the marker.

## Rough size

S–M: diagnosis first.

Contributor guide

Open the contributing guide

Research direction

Start at GenerateAutoscalerWorkerConfig and trace the deploy path that populates the kube-system/cluster-autoscaler-config Secret. Compare the consumed template with the expected marker and Longhorn-label behavior, then verify a newly provisioned autoscaler node and add a regression test for the actual autoscaler template.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.