ceph / ceph/ceph-csi

Race Condition During Concurrent Clone and Delete (CephFS & RBD)

Open
#6,321 2 comments 0 reactions 1 assignee Claimed by @nixpanic View on GitHub
component/cephfs component/rbd keepalive
Dominant language
Go
Stars
1.6k
Forks
617
Avg merge
5d 10h
Merged PRs (30d)
43

Description

The below comes from a **Copilot** suggestion, it applies to CephFS and RBD as well.

### Race Condition During Concurrent Clone and Delete:

#### Scenario:

- PVC-A (vol-123) exists and is mounted
- User creates PVC-B as clone of PVC-A
- User deletes PVC-A while clone is in progress

#### Without Source Volume Lock:

Thread A: CreateVolume(clone from vol-123)

- Locks: "pvc-b-name" (destination)
- Calls RBD CreateVolume (starts cloning)

Thread B: DeleteVolume(vol-123) - runs concurrently

- Locks: "vol-123" (different lock - both threads proceed!)
- Deletes NVMe-oF namespace → success
- Tries to delete RBD image → FAILS (has clones, RBD blocks it)
- Returns error

**Result: vol-123 has RBD image but NO namespace - PVC-A cannot be mounted until retry succeeds.**

#### Option 1: Add Source Volume Lock (prevents race)
```Go
// In CreateVolume, after extracting clone source:
if vol := contentSource.GetVolume(); vol != nil {
sourceVolumeID := vol.GetVolumeId()
if acquired := cs.volumeLocks.TryAcquire(sourceVolumeID); !acquired {
return nil, status.Errorf(codes.Aborted, "source volume operation in progress")
}
defer cs.volumeLocks.Release(sourceVolumeID)
}
```

- Pros: No broken state window, cleaner user experience
- Cons: Additional lock complexity, serializes operations

#### Option 2: Leave As-Is (rely on idempotency)

- DeleteNamespace is idempotent (returns nil on ENODEV)
- DeleteVolume will be retried by Kubernetes
- Eventually succeeds after clones are deleted
- Pros: Simpler code, no additional locks
- Cons: Temporary unpublishable state, more error logs during retries

Both approaches are valid - lock prevents the issue, idempotency recovers from it.

_Originally posted by @gadididi in https://github.com/ceph/ceph-csi/pull/6277#discussion_r3254221879_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.