Tokens and env vars stopped being injected

What happened:
A few days ago, we noticed that some operations that used to succeed on AWS from our pods started to receive 403s. After a lot of investigation, we noticed that the pods in some namespaces just stopped having their env variables and volumes changed with the relevant info and fell back into the machine credentials.

What you expected to happen:
I expected the injection to continue to work and work consistently across the namespaces as the policy was configured for all namespaces.

How to reproduce it (as minimally and precisely as possible):
I'm not exactly sure. I'd appreciate some help as it's happing in my cluster pretty consistently.

Anything else we need to know?:
Here is some grep showing the multiple namespaces (e.g. el, staging, ...) with different AWS env vars.

cm
    - name: AWS_REGION
    - name: AWS_REGION
    - name: AWS_REGION
demo
    - name: AWS_REGION
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      name: aws-iam-token
    - name: AWS_REGION
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      name: aws-iam-token
    - name: AWS_REGION
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      name: aws-iam-token
  - name: aws-iam-token
el
    - name: AWS_REGION
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      name: aws-iam-token
    - name: AWS_REGION
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      name: aws-iam-token
    - name: AWS_REGION
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      name: aws-iam-token
  - name: aws-iam-token
locked
    - name: AWS_REGION
    - name: AWS_REGION
    - name: AWS_REGION
staging
      - name: AWS_REGION
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        name: aws-iam-token
      - name: AWS_REGION
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        name: aws-iam-token
      - name: AWS_REGION
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        name: aws-iam-token
    - name: aws-iam-token
      - name: AWS_REGION
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        name: aws-iam-token
      - name: AWS_REGION
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        name: aws-iam-token
      - name: AWS_REGION
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        name: aws-iam-token
    - name: aws-iam-token
      - name: AWS_REGION
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        name: aws-iam-token
      - name: AWS_REGION
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        name: aws-iam-token
      - name: AWS_REGION
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::MYACCOUNTNUMBER:role/MYROLENAME
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        name: aws-iam-token
    - name: aws-iam-token

all of these were created with exactly the same yaml spec files.

Environment:

• AWS Region: us-east-1
• EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.4
• Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): 1.16
• Webhook Version: I'm not exactly sure as I think this runs within EKS itself.

Start by comparing the generated pod specs from the identical YAML spec files across the affected namespaces, then review the EKS cluster details from the mentioned aws eks describe-cluster commands and the webhook configuration/version. Done means identifying why injection differs between namespaces and restoring consistent environment-variable and volume injection without fallback credentials.