Race Condition During Concurrent Clone and Delete (CephFS & RBD)
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 617
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 43
Description
The below comes from a **Copilot** suggestion, it applies to CephFS and RBD as well.
### Race Condition During Concurrent Clone and Delete:
#### Scenario:
- PVC-A (vol-123) exists and is mounted
- User creates PVC-B as clone of PVC-A
- User deletes PVC-A while clone is in progress
#### Without Source Volume Lock:
Thread A: CreateVolume(clone from vol-123)
- Locks: "pvc-b-name" (destination)
- Calls RBD CreateVolume (starts cloning)
Thread B: DeleteVolume(vol-123) - runs concurrently
- Locks: "vol-123" (different lock - both threads proceed!)
- Deletes NVMe-oF namespace → success
- Tries to delete RBD image → FAILS (has clones, RBD blocks it)
- Returns error
**Result: vol-123 has RBD image but NO namespace - PVC-A cannot be mounted until retry succeeds.**
#### Option 1: Add Source Volume Lock (prevents race)
```Go
// In CreateVolume, after extracting clone source:
if vol := contentSource.GetVolume(); vol != nil {
sourceVolumeID := vol.GetVolumeId()
if acquired := cs.volumeLocks.TryAcquire(sourceVolumeID); !acquired {
return nil, status.Errorf(codes.Aborted, "source volume operation in progress")
}
defer cs.volumeLocks.Release(sourceVolumeID)
}
```
- Pros: No broken state window, cleaner user experience
- Cons: Additional lock complexity, serializes operations
#### Option 2: Leave As-Is (rely on idempotency)
- DeleteNamespace is idempotent (returns nil on ENODEV)
- DeleteVolume will be retried by Kubernetes
- Eventually succeeds after clones are deleted
- Pros: Simpler code, no additional locks
- Cons: Temporary unpublishable state, more error logs during retries
Both approaches are valid - lock prevents the issue, idempotency recovers from it.
_Originally posted by @gadididi in https://github.com/ceph/ceph-csi/pull/6277#discussion_r3254221879_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.