ceph / ceph/ceph-csi

Backport #6354 (remove snapshot lock from restore path) to release-v3.17

Open Beginner friendly
#6,516 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
1.6k
Forks
617
Avg merge
5d 10h
Merged PRs (30d)
43

Description

## What

PR #6354 (merged into `devel` on 2026-07-06) removes the `SnapshotLocks`
guard from `createVolumeFromSnapshot()`. v3.17.1 was released on 2026-08-24 —
seven weeks after the merge — but does not contain it: the lock is still
present in `internal/rbd/controllerserver.go` at the v3.17.1 tag.

Please consider cherry-picking it onto `release-v3.17`.

## Why it matters — measurements from a test cluster

Environment: ceph-csi v3.17.0, Ceph 10 NVMe OSDs / 24 TiB, Kubernetes with
KubeVirt; VMs clone their root disks from a per-image golden VolumeSnapshot.
external-provisioner v6.2.0 with `--worker-threads=100 --kube-api-qps=100`,
so neither sidecar concurrency nor API throttling is the limiter.

Batches of 120 × 10Gi block PVCs, `dataSource` = VolumeSnapshot:

| scenario | throughput |
|---|---|
| 120 PVCs, one source snapshot | **34.6/min** — completion curve perfectly linear, ~1.7 s per volume |
| 120 PVCs, four source snapshots (30 each) | **100/min** (~3×, near-linear) |
| `rbd clone` directly against the pool, sequentially, same source | **245/min** (244 ms/clone) |

During the two K8s runs the plugin logged 1,645 ×
`Aborted: an operation with the given Snapshot ID ... already exists`
(controllerserver.go:687). Because the guard is `TryAcquire`, losers are not
queued in-process but bounce back to csi-provisioner's exponential backoff;
under a burst this makes throughput *worse* than plain serialization — a
paced arrival stream reaches ~45/min from the same single snapshot, while a
one-shot burst of 120 only reaches 34.6/min.

The critical section holds far more than the clone itself: snapshot resolve
via omap, `librbd.CloneImage`, snapshot-metadata unset, encryption-config
copy, and expand — ~1.7 s of serial round trips where the clone itself needs
244 ms.

Net effect on v3.17: cloning many volumes from one snapshot — the standard
golden-image pattern for VM platforms — is capped at roughly 35 volumes/min
per source snapshot regardless of any configuration, and the Aborted retry
storm adds API and log noise on top.

## Notes

- #6322 (guard CreateVolume against concurrent DeleteVolume/DeleteSnapshot)
covers the race the lock was protecting; as noted in #6354 the restore
operation lock already covers the restore path.
- Happy to re-run the same benchmark against a release-v3.17 build with the
cherry-pick to confirm the numbers.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing PR #6354 and comparing internal/rbd/controllerserver.go on release-v3.17 with devel. Cherry-pick the change onto release-v3.17, run the repository’s relevant Go tests or build checks, and confirm the snapshot restore path no longer contains the lock while the release branch remains valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, release
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.