openshift / openshift/csi-operator
SMB CSI: CreateVolume succeeds after ~25s but csi-provisioner fails with DeadlineExceeded and no PV is created
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 25
- Forks
- 62
- Avg merge
- 8d 7h
- Merged PRs (30d)
- 11
Description
Environment
Platform: OpenShift
SMB CSI Driver Version: v1.19.1
StorageClass: xxx-sc
Provisioner: smb.csi.k8s.io
Workload: RabbitMQ StatefulSet (via RabbitMQ Operator)
Description
We're experiencing a provisioning issue with dynamic provision, where a PVC remains in Pending and no PV is created.
From the logs, the SMB CSI driver successfully processes the CreateVolume request and returns a valid response, but the external provisioner reports:
rpc error: code = DeadlineExceeded desc = context deadline exceeded
The provisioning process is then retried repeatedly and eventually starts returning:
rpc error: code = Aborted desc = An operation with the given Volume ID already exists
We were able to work around the issue by increasing the timeout used by the csi-provisioner sidecar by modifying the deployment manually, during the operation the operator was set on 0 replicas to dont make it reset our changes.
Symptoms
- PVC: persistence-rabbitmq-server-1 with status: Pending
- Pod scheduling fails with: ### 0/X nodes are available: pod has unbound immediate PersistentVolumeClaims
- No corresponding PV is created.
PVC Events
Normal Provisioning External provisioner is provisioning volume for claim "prod/persistence-rabbitmq-server-1"
Warning ProvisioningFailed rpc error: code = DeadlineExceeded desc = context deadline exceeded
Warning ProvisioningFailed rpc error: code = Aborted desc = An operation with the given Volume ID already exists
Driver Logs
The SMB CSI driver receives the request correctly:
GRPC call: /csi.v1.Controller/CreateVolume
Request:
{
"name":"pvc-3ca08688-1234",
"parameters":{
"csi.storage.k8s.io/pvc/name":"persistence-rabbitmq-server-1",
"csi.storage.k8s.io/pvc/namespace":"prod",
"source":"xxx.com/yyy/zzz"
}
}
The share is mounted successfully:
mount "//xxx.com/yyy/zzz" on "/tmp/pvc-3ca08688-1234" succeeded
A successful response is eventually returned:
{
"volume": {
"volume_id": "xxx.com/yyy/zzz#pvc-3ca08688-0491-4321"
}
}
Timing Analysis
The provisioning operation appears to take approximately 20-30 seconds.
Example:
09:01:08 CreateVolume request received
09:01:18 DeadlineExceeded reported by provisioner
09:01:33 CreateVolume response returned successfully
Another example:
09:02:42 CreateVolume request received
09:03:07 CreateVolume response returned successfully
This seems to indicate that the driver eventually completes successfully, but the provisioner has already timed out.
Impact
The result is:
PVC -> Pending
PV -> Not created
Pod -> Pending
Provisioning is retried indefinitely.
Manual Workaround
We stopped the SMB CSI operator reconciliation and increased the timeout used by the csi-provisioner container, by adding the following option in the container definition inside smb-csi-driver-controller deployment:
args:
- '--timeout=60s'
After increasing the timeout:
PVC -> Bound
PV -> Created
Pod -> Running
Provisioning completed successfully without any other changes.
Question
Is there a supported way to configure the timeout used by the SMB CSI provisioner?
We checked the following object:
kind: ClusterCSIDriver
metadata:
name: smb.csi.k8s.io
but could not find any documented configuration option for the provisioner timeout.
Spec currently exposes:
...
spec:
driverConfig:
logLevel:
managementState:
observedConfig:
operatorLogLevel:
storageClassState:
unsupportedConfigOverrides:
Is there a supported configuration mechanism for increasing the CreateVolume timeout?
Additional Notes
The issue is reproducible and disappears consistently after increasing the provisioner timeout.
The SMB share itself is reachable and functional, and the driver successfully mounts and validates the share during provisioning. The failure seems to occur because the provisioning operation takes longer than the timeout expected by the provisioner sidecar.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the ClusterCSIDriver spec and the reconciliation that manages the smb-csi-driver-controller deployment, then reproduce the CreateVolume timing described in the issue. Determine whether the csi-provisioner timeout has a supported configuration path; done means the timeout can be configured without manual reconciliation changes and the PVC creates its PV successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100