openshift / openshift/csi-operator

SMB CSI: CreateVolume succeeds after ~25s but csi-provisioner fails with DeadlineExceeded and no PV is created

Open
#624 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
25
Forks
62
Avg merge
8d 7h
Merged PRs (30d)
11

Description

Environment

Platform: OpenShift
SMB CSI Driver Version: v1.19.1
StorageClass: xxx-sc
Provisioner: smb.csi.k8s.io
Workload: RabbitMQ StatefulSet (via RabbitMQ Operator)

Description

We're experiencing a provisioning issue with dynamic provision, where a PVC remains in Pending and no PV is created.
From the logs, the SMB CSI driver successfully processes the CreateVolume request and returns a valid response, but the external provisioner reports:

rpc error: code = DeadlineExceeded desc = context deadline exceeded

The provisioning process is then retried repeatedly and eventually starts returning:

rpc error: code = Aborted desc = An operation with the given Volume ID already exists

We were able to work around the issue by increasing the timeout used by the csi-provisioner sidecar by modifying the deployment manually, during the operation the operator was set on 0 replicas to dont make it reset our changes.

Symptoms
  • PVC: persistence-rabbitmq-server-1 with status: Pending
  • Pod scheduling fails with: ### 0/X nodes are available: pod has unbound immediate PersistentVolumeClaims
  • No corresponding PV is created.
PVC Events

Normal Provisioning External provisioner is provisioning volume for claim "prod/persistence-rabbitmq-server-1"
Warning ProvisioningFailed rpc error: code = DeadlineExceeded desc = context deadline exceeded
Warning ProvisioningFailed rpc error: code = Aborted desc = An operation with the given Volume ID already exists

Driver Logs

The SMB CSI driver receives the request correctly:

GRPC call: /csi.v1.Controller/CreateVolume

Request:

{
"name":"pvc-3ca08688-1234",
"parameters":{
"csi.storage.k8s.io/pvc/name":"persistence-rabbitmq-server-1",
"csi.storage.k8s.io/pvc/namespace":"prod",
"source":"xxx.com/yyy/zzz"
}
}

The share is mounted successfully:

mount "//xxx.com/yyy/zzz" on "/tmp/pvc-3ca08688-1234" succeeded

A successful response is eventually returned:

{
"volume": {
"volume_id": "xxx.com/yyy/zzz#pvc-3ca08688-0491-4321"
}
}

Timing Analysis

The provisioning operation appears to take approximately 20-30 seconds.

Example:

09:01:08 CreateVolume request received
09:01:18 DeadlineExceeded reported by provisioner
09:01:33 CreateVolume response returned successfully

Another example:

09:02:42 CreateVolume request received
09:03:07 CreateVolume response returned successfully

This seems to indicate that the driver eventually completes successfully, but the provisioner has already timed out.

Impact

The result is:

PVC -> Pending
PV -> Not created
Pod -> Pending

Provisioning is retried indefinitely.

Manual Workaround

We stopped the SMB CSI operator reconciliation and increased the timeout used by the csi-provisioner container, by adding the following option in the container definition inside smb-csi-driver-controller deployment:

args:
    - '--timeout=60s'

After increasing the timeout:

PVC -> Bound
PV -> Created
Pod -> Running

Provisioning completed successfully without any other changes.

Question

Is there a supported way to configure the timeout used by the SMB CSI provisioner?

We checked the following object:

kind: ClusterCSIDriver
metadata:
name: smb.csi.k8s.io

but could not find any documented configuration option for the provisioner timeout.

Spec currently exposes:

...
spec:
  driverConfig:
  logLevel:
  managementState:
  observedConfig:
  operatorLogLevel:
  storageClassState:
  unsupportedConfigOverrides:

Is there a supported configuration mechanism for increasing the CreateVolume timeout?

Additional Notes

The issue is reproducible and disappears consistently after increasing the provisioner timeout.

The SMB share itself is reachable and functional, and the driver successfully mounts and validates the share during provisioning. The failure seems to occur because the provisioning operation takes longer than the timeout expected by the provisioner sidecar.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ClusterCSIDriver spec and the reconciliation that manages the smb-csi-driver-controller deployment, then reproduce the CreateVolume timing described in the issue. Determine whether the csi-provisioner timeout has a supported configuration path; done means the timeout can be configured without manual reconciliation changes and the PVC creates its PV successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.