oracle / oracle/oci-cloud-controller-manager

oci-bv - Timed out waiting for backup to become available

Open
#491 4 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
158
Forks
108
PR merge metrics
No merged PRs in 30d

Description

Hi,

We are using a VolumeSnapshotClass as below for Block volume snapshotting:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: oci-bv-snapshot-incremental
driver: blockvolume.csi.oraclecloud.com
parameters:
  backupType: incremental # No functional restore difference between full and incremental
deletionPolicy: Delete

This is integrated with CNPG for lower environment database volume snapshots.
Occasionally (every few weeks), we find these backups failing. with the error:

DeadlineExceeded desc = Timed out waiting for backup to become available

It looks like this is being thrown by the oci-bv csi here:
https://github.com/oracle/oci-cloud-controller-manager/blob/411bfeb22d242ba4c1f8b647884f507869968a2f/pkg/csi/driver/bv_controller.go#L1099

Which uses a timeout of 45 seconds as defined here:
https://github.com/oracle/oci-cloud-controller-manager/blob/411bfeb22d242ba4c1f8b647884f507869968a2f/pkg/csi/driver/bv_controller.go#L1068

However, in practice a 45 second timeout is too conservative, looking in the logs, we see the following times for snapshot creation in uk-london-1 between going from com.oraclecloud.BlockVolumes.CreateVolumeBackup.begin to com.oraclecloud.BlockVolumes.CreateVolumeBackup.end state.

Over 9 samples: average: 37.4 seconds | min: 34 seconds | max: 41 seconds

With a backupPollInterval of 5 seconds, the CSI steps just outside of the permissible timeout of 45 seconds.
https://github.com/oracle/oci-cloud-controller-manager/blob/master/pkg/oci/client/block_storage.go#L150C36-L150C60

https://github.com/oracle/oci-cloud-controller-manager/blob/master/pkg/oci/client/block_storage.go#L42

I believe the solution for this would be to increase the available timeout to 60 seconds to align better with the expected response times from the API.
https://github.com/oracle/oci-cloud-controller-manager/blob/411bfeb22d242ba4c1f8b647884f507869968a2f/pkg/csi/driver/bv_controller.go#L1068

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/csi/driver/bv_controller.go around the 45-second backup timeout, then read the backup polling configuration in pkg/oci/client/block_storage.go. Confirm the timeout and polling behavior against the reported creation times; done means backup creation no longer fails prematurely for these expected API response times.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.