Multiple CredentialsRequest with same secretRef causes CCO to delete and create AWS access keys every 70 minutes

Problem Statement

Multiple CredentialsRequest (for example, CR1 and CR2) with same secretRef.namespace and secretRef.name causes CCO to mix up which AWS accessKeyID belongs to which CR. When it cannot find an AWS accessKeyID it believes belongs to CR1, (it actually was created by CR2 with the same secretRef values as CR1) it deletes all access keys belonging to CR1 and generates new ones. CR2 overwrites the access keys for CR1 again, and the cycle continues forever. The deleting and creating of AWS access keys seems to happen once every 70 minutes.

Notice in the logs below, CCO creates accessKeyID=*QYV5 for cr=cockroachdb/cr-s3-sync. Later it warns "access key no longer exists" accessKeyID=*QYV5 but it's checking cr=cockroachdb/cr-s3-cockroachdb-backups, not cr=cockroachdb/cr-s3-sync, for which the accessKeyID was originally created.

This mix up results in the bad behavior.

Logs

time="2024-11-26T17:15:50Z" level=info msg="all access keys deleted" actuator=aws cr=cockroachdb/cr-s3-sync
time="2024-11-26T17:15:50Z" level=info msg="access key created" accessKeyID=*QYV5 actuator=aws cr=cockroachdb/cr-s3-sync
time="2024-11-26T17:15:50Z" level=info msg="processing secret" actuator=aws cr=cockroachdb/cr-s3-sync targetSecret=cockroachdb/s3-cockroachdb-backups
time="2024-11-26T17:15:50Z" level=info msg="processed secret" actuator=aws cr=cockroachdb/cr-s3-sync operation=updated targetSecret=cockroachdb/s3-cockroachdb-backups
time="2024-11-26T17:15:50Z" level=info msg="status has changed, updating" controller=credreq cr=cockroachdb/cr-s3-sync secret=cockroachdb/s3-cockroachdb-backups
time="2024-11-26T17:15:51Z" level=info msg="syncing credentials request" controller=credreq cr=cockroachdb/cr-s3-cockroachdb-backups
time="2024-11-26T17:15:51Z" level=warning msg="access key no longer exists" accessKeyID=*QYV5 actuator=aws cr=cockroachdb/cr-s3-cockroachdb-backups
time="2024-11-26T17:15:51Z" level=info msg="user exists" actuator=aws cr=cockroachdb/cr-s3-cockroachdb-backups userName=vapo-aws-valip-stg-m-cr-s3-cockroachdb-backups-gj7pr
time="2024-11-26T17:15:51Z" level=warning msg="access key no longer exists" accessKeyID=*QYV5 actuator=aws cr=cockroachdb/cr-s3-cockroachdb-backups
time="2024-11-26T17:15:51Z" level=info msg="generating new AWS access key" actuator=aws cr=cockroachdb/cr-s3-cockroachdb-backups
time="2024-11-26T17:15:51Z" level=info msg="deleting all AWS access keys" actuator=aws cr=cockroachdb/cr-s3-cockroachdb-backups
time="2024-11-26T17:15:51Z" level=info msg="deleting access key" accessKeyID=*F6DW actuator=aws cr=cockroachdb/cr-s3-cockroachdb-backups
time="2024-11-26T17:15:51Z" level=info msg="all access keys deleted" actuator=aws cr=cockroachdb/cr-s3-cockroachdb-backups
time="2024-11-26T17:15:51Z" level=info msg="access key created" accessKeyID=*WVTP actuator=aws cr=cockroachdb/cr-s3-cockroachdb-backups
time="2024-11-26T17:15:51Z" level=info msg="processing secret" actuator=aws cr=cockroachdb/cr-s3-cockroachdb-backups targetSecret=cockroachdb/s3-cockroachdb-backups
time="2024-11-26T17:15:51Z" level=info msg="reconciling clusteroperator status"
time="2024-11-26T17:15:51Z" level=info msg="processed secret" actuator=aws cr=cockroachdb/cr-s3-cockroachdb-backups operation=updated targetSecret=cockroachdb/s3-cockroachdb-backups

Potential Solution

A simple solution to this problem would be to not allow multiple CredentialsRequest objects to be created with the same secretRef.namespace and secretRef.name values. CCO could check for this condition (maybe via a validating webhook?) when processing new CredentialsRequest objects.

Start by tracing CredentialsRequest reconciliation and the AWS access-key ownership handling. Reproduce the issue with two CredentialsRequest objects sharing the same secretRef.namespace and secretRef.name, then inspect the relevant tests or add a focused reproduction. Done means duplicate references are rejected or reconciled without repeatedly deleting and recreating access keys.