Backport #6354 (remove snapshot lock from restore path) to release-v3.17
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 617
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 43
Description
## What
PR #6354 (merged into `devel` on 2026-07-06) removes the `SnapshotLocks`
guard from `createVolumeFromSnapshot()`. v3.17.1 was released on 2026-08-24 —
seven weeks after the merge — but does not contain it: the lock is still
present in `internal/rbd/controllerserver.go` at the v3.17.1 tag.
Please consider cherry-picking it onto `release-v3.17`.
## Why it matters — measurements from a test cluster
Environment: ceph-csi v3.17.0, Ceph 10 NVMe OSDs / 24 TiB, Kubernetes with
KubeVirt; VMs clone their root disks from a per-image golden VolumeSnapshot.
external-provisioner v6.2.0 with `--worker-threads=100 --kube-api-qps=100`,
so neither sidecar concurrency nor API throttling is the limiter.
Batches of 120 × 10Gi block PVCs, `dataSource` = VolumeSnapshot:
| scenario | throughput |
|---|---|
| 120 PVCs, one source snapshot | **34.6/min** — completion curve perfectly linear, ~1.7 s per volume |
| 120 PVCs, four source snapshots (30 each) | **100/min** (~3×, near-linear) |
| `rbd clone` directly against the pool, sequentially, same source | **245/min** (244 ms/clone) |
During the two K8s runs the plugin logged 1,645 ×
`Aborted: an operation with the given Snapshot ID ... already exists`
(controllerserver.go:687). Because the guard is `TryAcquire`, losers are not
queued in-process but bounce back to csi-provisioner's exponential backoff;
under a burst this makes throughput *worse* than plain serialization — a
paced arrival stream reaches ~45/min from the same single snapshot, while a
one-shot burst of 120 only reaches 34.6/min.
The critical section holds far more than the clone itself: snapshot resolve
via omap, `librbd.CloneImage`, snapshot-metadata unset, encryption-config
copy, and expand — ~1.7 s of serial round trips where the clone itself needs
244 ms.
Net effect on v3.17: cloning many volumes from one snapshot — the standard
golden-image pattern for VM platforms — is capped at roughly 35 volumes/min
per source snapshot regardless of any configuration, and the Aborted retry
storm adds API and log noise on top.
## Notes
- #6322 (guard CreateVolume against concurrent DeleteVolume/DeleteSnapshot)
covers the race the lock was protecting; as noted in #6354 the restore
operation lock already covers the restore path.
- Happy to re-run the same benchmark against a release-v3.17 build with the
cherry-pick to confirm the numbers.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing PR #6354 and comparing internal/rbd/controllerserver.go on release-v3.17 with devel. Cherry-pick the change onto release-v3.17, run the repository’s relevant Go tests or build checks, and confirm the snapshot restore path no longer contains the lock while the release branch remains valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, release
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100