AWS IAM Role for Service Account support

We're trying to use vitess-operator with an S3 backup spec defined in our VitessCluster manifest. Example as follows:

spec:  
  backup:
    engine: xtrabackup
    locations:
      - s3:
          bucket: my-s3-bucket
          region: us-east-1

From the logs, it appears that vttablet attempts to read the backup S3 bucket to see it it needs to restore from the latest backup. As such, we need to provide AWS credentials to allow the pod access to read the S3 bucket.

Typically, we use IAM Roles for Service Accounts (IRSA) which allows annotating a Service Account with an IAM Role arn: https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html

• Using IRSA, when a pod is created using this annotated service account, AWS_* environment variables are added to the pod so the pod can authenticate and use the the IAM role.

The issue here is that vitess-operator seems to see these AWS_ environment variables on the pod and attempts to $patch: delete these variables.

Example log taken from vitess-operator:

time="2023-02-14T22:12:16Z" level=info msg="Updating object in place" diff="metadata:\n  annotations:\n    planetscale.com/observed-shard-generation: \"1\"\n    rollout.planetscale.com/scheduled: |\n      spec:\n        containers:\n        - $setElementOrder/env:\n          - name: VTROOT\n          - name: VTDATAROOT\n          - name: VT_MYSQL_ROOT\n          - name: MYSQL_FLAVOR\n          - name: EXTRA_MY_CNF\n          - name: POD_IP\n          env:\n          - $patch: delete\n            name: AWS_DEFAULT_REGION\n          - $patch: delete\n            name: AWS_REGION\n          - $patch: delete\n            name: AWS_ROLE_ARN\n          - $patch: delete\n            name: AWS_STS_REGIONAL_ENDPOINTS\n          - $patch: delete\n            name: AWS_WEB_IDENTITY_TOKEN_FILE\n          name: vttablet\n        - $setElementOrder/env:\n          - name: VTROOT\n          - name: VTDATAROOT\n          - name: VT_MYSQL_ROOT\n          - name: MYSQL_FLAVOR\n          - name: EXTRA_MY_CNF\n          - name: POD_IP\n          env:\n          - $patch: delete\n            name: AWS_DEFAULT_REGION\n          - $patch: delete\n            name: AWS_REGION\n          - $patch: delete\n            name: AWS_ROLE_ARN\n          - $patch: delete\n            name: AWS_STS_REGIONAL_ENDPOINTS\n          - $patch: delete\n            name: AWS_WEB_IDENTITY_TOKEN_FILE\n          name: mysqld\n        - $setElementOrder/env:\n          - name: DATA_SOURCE_NAME\n          env:\n          - $patch: delete\n            name: AWS_DEFAULT_REGION\n          - $patch: delete\n            name: AWS_REGION\n          - $patch: delete\n            name: AWS_ROLE_ARN\n          - $patch: delete\n            name: AWS_STS_REGIONAL_ENDPOINTS\n          - $patch: delete\n            name: AWS_WEB_IDENTITY_TOKEN_FILE\n          name: mysqld-exporter\n        initContainers:\n        - env: null\n          name: init-vt-root\n        - env: null\n          name: init-mysql-socket\n" gvk="/v1, Kind=Pod" key=dev/dev-vitess-vttablet-useast1a-OMITTED

Is there any way to force vitess-operator to ignore the AWS_* environment variable discrepancies?

For context, these tests were performed on an AWS-hosted EKS cluster using vitess-operator v2.9.0-rc1

Start with the VitessCluster backup configuration and the vttablet pod update shown in the operator log, then trace the reconciliation logic that compares pod environment variables. Check how AWS_* variables injected by an annotated service account are handled, and verify that the completed change preserves them while still applying operator-managed updates.