oracle / oracle/oci-cloud-controller-manager
oci-bv - Timed out waiting for backup to become available
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 158
- Forks
- 108
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
We are using a VolumeSnapshotClass as below for Block volume snapshotting:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: oci-bv-snapshot-incremental
driver: blockvolume.csi.oraclecloud.com
parameters:
backupType: incremental # No functional restore difference between full and incremental
deletionPolicy: Delete
This is integrated with CNPG for lower environment database volume snapshots.
Occasionally (every few weeks), we find these backups failing. with the error:
DeadlineExceeded desc = Timed out waiting for backup to become available
It looks like this is being thrown by the oci-bv csi here:
https://github.com/oracle/oci-cloud-controller-manager/blob/411bfeb22d242ba4c1f8b647884f507869968a2f/pkg/csi/driver/bv_controller.go#L1099
Which uses a timeout of 45 seconds as defined here:
https://github.com/oracle/oci-cloud-controller-manager/blob/411bfeb22d242ba4c1f8b647884f507869968a2f/pkg/csi/driver/bv_controller.go#L1068
However, in practice a 45 second timeout is too conservative, looking in the logs, we see the following times for snapshot creation in uk-london-1 between going from com.oraclecloud.BlockVolumes.CreateVolumeBackup.begin to com.oraclecloud.BlockVolumes.CreateVolumeBackup.end state.
Over 9 samples: average: 37.4 seconds | min: 34 seconds | max: 41 seconds
With a backupPollInterval of 5 seconds, the CSI steps just outside of the permissible timeout of 45 seconds.
https://github.com/oracle/oci-cloud-controller-manager/blob/master/pkg/oci/client/block_storage.go#L150C36-L150C60
I believe the solution for this would be to increase the available timeout to 60 seconds to align better with the expected response times from the API.
https://github.com/oracle/oci-cloud-controller-manager/blob/411bfeb22d242ba4c1f8b647884f507869968a2f/pkg/csi/driver/bv_controller.go#L1068
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pkg/csi/driver/bv_controller.go around the 45-second backup timeout, then read the backup polling configuration in pkg/oci/client/block_storage.go. Confirm the timeout and polling behavior against the reported creation times; done means backup creation no longer fails prematurely for these expected API response times.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100