Azure / Azure/local-csi-driver

Noisy logs due to non-owning nodes emitting `FailedPrecondition` ERRORs + PV Warning events for deletes the owning node already handled

Open
#528 0 comments 1 reaction 1 assignee Claimed by @jmclong View on GitHub
Dominant language
Go
Stars
33
Forks
16
Avg merge
10h 17m
Merged PRs (30d)
28

Description

TLDR: Currently the happy path for volume deletion, where an NVMe volume is deleted successfully by the node that owns it, produces a large volume of error logs. The volume of logs scales with the number of nodes in the pool.

Ideally, if the owning node still exists, non-owning nodes should recognise that they have no work to do and that nothing is in an unexpected state - silently.

## What happens

On a standard Azure Container Storage v2 install for local NVMe, non-replicated volumes always use the hyperconverged webhook (this is the default and, for non-replicated volumes, is not configurable). With the webhook active, PVs carry no node affinity, so **all** DaemonSet driver instances process every `DeleteVolume` request. If I understand `docs/design/pv-cleanup.md` correctly, this is by design as it enables the overall system to recover from failed nodes. Great!

[docs/design/pv-cleanup.md](https://github.com/Azure/local-csi-driver/blob/787bcff3f7e0a66f83725f8591e37f42db5e865d/docs/design/pv-cleanup.md) says a non-owning instance should check whether the node is deleted and return success if so. But in the normal case, where the owning node still exists and successfully deletes the volume itself, the other instances currently fall through to `FailedPrecondition: Volume is on a different node`. Each logs this as an ERROR, writes a `VolumeFailedDelete` **Warning event onto the PV object**, logs an `Unhandled Error`, and retries with backoff.

The volume is deleted correctly. This is just a log/event noise issue, but it scales with cluster size and occurs on the default install path - so it seems worth raising and addressing!

## Evidence (log lines involving a single PVC, 65-node pool, 107s window)

| Metric | Value |
| --- | --- |
| Total log lines | **1,567** |
| ERROR-level lines | **768** |
| Owning node lines (correct/functional lifecycle) | 31 |
| Other 64 nodes | 1,536 |
| `VolumeFailedDelete` events on the PV | 256 |

Each non-owning node emits ~24 lines (`failures=0..3`) in ~1.4s. Per-volume line count is uniform across ~600 volumes (~127–328 each): fixed per-node cost × node count.

Representative non-owning node:

```
[ERROR] "Volume deletion failed" err="... FailedPrecondition ... Volume is on a different node "
[ERROR] "Event occurred" ... type="Warning" reason="VolumeFailedDelete"
[ERROR] "Unhandled Error" err="error syncing volume ... FailedPrecondition ..."
"Retrying syncing volume" failures=0..3
```

## Environment

- Azure Container Storage v2 (`microsoft.azurecontainerstoragev2` cluster extension — the documented Terraform/CLI install), local NVMe storage class
- `local-csi-driver:v0.2.17`, csi-provisioner `v5.2.0`, `--node-deployment --strict-topology=true`
- AKS, 65-node pool, generic ephemeral (non-replicated) volumes → hyperconverged webhook active by default
- Confirmed on PVs: `.spec.nodeAffinity` unset, `.spec.csi.volumeAttributes."localdisk.csi.acstor.io/selected-initial-node"` populated
- Workload: short-lived CI build agents → high volume churn

## Suggested fix

Extend the non-owning-node handling described in `pv-cleanup.md` so an instance that doesn't own the volume returns cleanly in **both** cases: node deleted (already designed) **and** node present / already handled by the owner — without logging `FailedPrecondition` at ERROR or emitting a `VolumeFailedDelete` event. Since this is the default local-NVMe path, it affects any cluster following the standard install.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.