linode / linode/apl-core

Block Storage Volumes are unattributable in the Cloud Manager, and the obvious fix breaks them

Open
#3,607 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go Template
Stars
2.3k
Forks
186
Avg merge
3d 11h
Merged PRs (30d)
66

Description

Platform-impact companion to **linode/linode-blockstorage-csi-driver#603**, which carries the technical request. Filing here because APL users hit this, and because the workaround is dangerous enough to be worth naming explicitly.

### The problem for APL operators

Every Block Storage Volume APL provisions is labelled `pvc-` in the Cloud Manager. There is no way to tell which workload owns one without joining back through Kubernetes — and once the cluster is deleted, no way at all. A leaked Volume on a dead cluster is anonymous.

That affects day-to-day cost attribution, the quota census, and cleanup after a failed cluster.

### The trap — please do not "fix" this by renaming

Renaming the Volume to something readable is the obvious workaround and it **permanently breaks the Volume**. The CSI resolves the block device by label, from the label baked into the PV's immutable `volumeHandle`. Rename it and the driver hunts a device path that no longer exists:

```
MountVolume.MountDevice failed for volume "pvc-72af5c8ff02c4813":
Unable to find device path out of attempted paths:
[/dev/disk/by-id/linode-pvc-72af5c8ff02c4813
/dev/disk/by-id/scsi-0Linode_Volume_pvc-72af5c8ff02c4813]
```

Worse, it is a **deferred** failure. A renamed Volume that is already mounted keeps working, and fails on its next attach — a node drain, a reschedule, an upgrade. We ran a reconciler that did exactly this. It reproduced 3/3 on fresh clusters, and on a long-lived one presented as "1 of 3 Loki ingesters ready for 16 days" — a partial outage nobody attributed to storage. We have retired it; there is no safe subset.

If any APL component or downstream distribution renames CSI-provisioned Volumes, it has this bug.

### What APL could do

1. **Enable `--extra-create-metadata`** on the external-provisioner APL ships or depends on. It is the precondition for #603: without it the driver never receives `csi.storage.k8s.io/pvc/name` and `/pvc/namespace`, so the fix cannot work even once merged. On managed LKE-E this is not operator-settable, which is why it needs to come from the platform.
2. **Carry the request upstream** — #603 is small and safe by construction, because `CreateVolume` builds the handle from the Volume it just created, so label and handle cannot diverge.
3. **Consider Volume tags in the meantime.** Tags are not part of any device path, so they can be written and corrected freely. A per-workload tag gives back at-a-glance identity in the Cloud Manager with none of the risk above. This is the direction we are taking downstream.

Happy to supply the reproduction detail, the reconciler logs correlating the rename against the failed mount, or review of a fix.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the external-provisioner configuration APL ships or depends on, then read linode/linode-blockstorage-csi-driver#603 and the documented device-path failure. Confirm the chosen platform direction—metadata support, carrying the upstream request, or Volume tags—against the reproduction details; done means workload identity is recoverable without renaming CSI-provisioned Volumes.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.