Vault agent container is not getting added to client application pod

After successful deployment of vault and consul to same namespace, nginx application is deployed to test vault agent injectior. Though correct annotations are added, We can see ngnix pod as 1/1. Vault-agent addon in nginx pod is not visible.
Steps to reproduce the behavior:
1. Install and vault and consul using a single helmchart
2. Unseal and initialize vault
3. Deploy nginx application using service DNS/FQDN, required role and policies in the same namespace

Logs/deployment yamls
Injector logs
api-server logs



kubectl get deploy eops-vault-eccs-vault-helmcharts-agent-injector -n eops-dependencies -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
meta.helm.sh/release-name: eops-vault
meta.helm.sh/release-namespace: eops-dependencies
creationTimestamp: "2022-07-29T17:18:50Z"
generation: 1
labels:
app.kubernetes.io/instance: eops-vault
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: eccs-vault-helmcharts-agent-injector
component: webhook
name: eops-vault-eccs-vault-helmcharts-agent-injector
namespace: eops-dependencies
resourceVersion: "7394620"
uid: 5843e26c-7140-4299-b4e3-f7c85ccded51
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: eops-vault
app.kubernetes.io/name: eccs-vault-helmcharts-agent-injector
component: webhook
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/instance: eops-vault
app.kubernetes.io/name: eccs-vault-helmcharts-agent-injector
component: webhook
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: eops-vault
app.kubernetes.io/name: eccs-vault-helmcharts-agent-injector
component: webhook
topologyKey: kubernetes.io/hostname
containers:
- args:
- agent-inject
- 2>&1
env:
- name: AGENT_INJECT_LISTEN
value: :8080
- name: AGENT_INJECT_LOG_LEVEL
value: info
- name: AGENT_INJECT_VAULT_ADDR
value: https://xx.xx.x.svc:8200
- name: AGENT_INJECT_VAULT_AUTH_PATH
value: auth/kubernetes
- name: AGENT_INJECT_VAULT_IMAGE
value: xxx.xx.xx/eccs-vault-dockerimages:vault_1.9.0_0.18.0
- name: AGENT_INJECT_TLS_CERT_FILE
value: /etc/webhook/certs/vault.crt
- name: AGENT_INJECT_TLS_KEY_FILE
value: /etc/webhook/certs/vault.key
- name: AGENT_INJECT_LOG_FORMAT
value: standard
- name: AGENT_INJECT_REVOKE_ON_SHUTDOWN
value: "false"
- name: AGENT_INJECT_CPU_REQUEST
value: 250m
- name: AGENT_INJECT_CPU_LIMIT
value: 500m
- name: AGENT_INJECT_MEM_REQUEST
value: 64Mi
- name: AGENT_INJECT_MEM_LIMIT
value: 128Mi
- name: AGENT_INJECT_DEFAULT_TEMPLATE
value: map
- name: AGENT_INJECT_TEMPLATE_CONFIG_EXIT_ON_RETRY_FAILURE
value: "true"
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: xxx.xx.xx/eccs-vault-dockerimages:vault-k8s_0.14.1_0.18.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 2
httpGet:
path: /health/ready
port: 8080
scheme: HTTPS
initialDelaySeconds: 5
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 5
name: sidecar-injector
readinessProbe:
failureThreshold: 2
httpGet:
path: /health/ready
port: 8080
scheme: HTTPS
initialDelaySeconds: 5
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: 250m
memory: 256Mi
requests:
cpu: 250m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/webhook/certs
name: webhook-certs
readOnly: true
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: docker-image
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 100
serviceAccount: eops-vault-eccs-vault-helmcharts-agent-injector
serviceAccountName: eops-vault-eccs-vault-helmcharts-agent-injector
terminationGracePeriodSeconds: 30
volumes:
- name: webhook-certs
secret:
defaultMode: 420
secretName: vault-server-tls
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2022-07-29T17:18:50Z"
lastUpdateTime: "2022-07-29T17:18:57Z"
message: ReplicaSet "eops-vault-eccs-vault-helmcharts-agent-injector-5df67b6866"
has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: "2022-08-05T08:01:14Z"
lastUpdateTime: "2022-08-05T08:01:14Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1

apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: "2022-08-03T11:40:29Z"
generation: 1
name: app-example-deployment1
namespace: eops-dependencies
resourceVersion: "7395004"
uid: a5a34b00-c638-457e-8a7b-07a939fda7d8
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: app-example1
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
vault.hashicorp.com/agent-cache-enable: "true"
vault.hashicorp.com/agent-cache-use-auto-auth-token: "true"
vault.hashicorp.com/agent-configmap: my-configmap
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-command-db-creds: sh -c 'kill -HUP $(pidof
client)'
vault.hashicorp.com/agent-inject-status: update
vault.hashicorp.com/agent-inject-token: "true"
vault.hashicorp.com/log-level: debug
vault.hashicorp.com/tls-secret: vault-server-tls
creationTimestamp: null
labels:
app: app-example1
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: app
resources:
limits:
cpu: 250m
memory: 256Mi
requests:
cpu: 250m
memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2022-08-03T11:40:29Z"
lastUpdateTime: "2022-08-03T11:40:49Z"
message: ReplicaSet "app-example-deployment1-fdb6988b8" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: "2022-08-05T08:01:55Z"
lastUpdateTime: "2022-08-05T08:01:55Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1

Client application pod, nginx pod in this case should have 2 container nginx and vault-agent conatiner.

**Environment**
* Kubernetes version: v1.21.5-gke.1200
* Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): GKE - Anthos
* vault-helm version: 0.18.0

Chart values:
Chart.yaml
apiVersion: v2
appVersion: 1.9.0
dependencies:
- name: consul
repository: ""
version: 0.39.0
description: Official HashiCorp Vault Chart
home: https://www.vaultproject.io
icon: https://github.com/hashicorp/vault/raw/f22d202cde2018f9455dec755118a9b84586e082/Vault_PrimaryLogo_Black.png
keywords:
- vault
- security
- encryption
- secrets
- management
- automation
- infrastructure
kubeVersion: '>= 1.14.0-0'
name: eccs-vault-helmcharts
sources:
- https://github.com/hashicorp/vault
- https://github.com/hashicorp/vault-helm
- https://github.com/hashicorp/vault-k8s
- https://github.com/hashicorp/vault-csi-provider
version: 0.18.0

Values.yaml
# Available parameters and their default values for the Vault chart.

global:
# enabled is the master enabled switch. Setting this to true or false
# will enable or disable all the components within this chart by default.
enabled: true
# Image pull secret to use for registry authentication.
# Alternatively, the value may be specified as an array of strings.
imagePullSecrets:
# imagePullSecrets:
- name: docker-image
# TLS for end-to-end encrypted transport
tlsDisable: false
# If deploying to OpenShift
openshift: false
# enable monitoring to create dashboard in grafana
monitoring:
enable: true
label: "eccs-central-grafana-grafana"
# enable alerting to notify about vault status using prometheus alertmanager
alerting:
enable: true
label: "eccs-central-prometheus-prometheus"
# Create PodSecurityPolicy for pods
psp:
enable: false
# Annotation for PodSecurityPolicy.
# This is a multi-line templated string map, and can also be set as YAML.
annotations: |
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default,runtime/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default

injector:
# True if you want to enable vault agent injection.
enabled: true

replicas: 1

# Configures the port the injector should listen on
port: 8080

# If multiple replicas are specified, by default a leader will be determined
# so that only one injector attempts to create TLS certificates.
leaderElector:
enabled: true

# If true, will enable a node exporter metrics endpoint at /metrics.
metrics:
enabled: false

# External vault server address for the injector to use. Setting this will
# disable deployment of a vault server along with the injector.
externalVaultAddr: ""

# image sets the repo and tag of the vault-k8s image to use for the injector.
image:
repository: "xxx.xx.x/eccs-vault-dockerimages"
tag: "vault-k8s_0.14.1_0.18.0"
pullPolicy: IfNotPresent

# agentImage sets the repo and tag of the Vault image to use for the Vault Agent
# containers. This should be set to the official Vault image. Vault 1.3.1+ is
# required.
agentImage:
repository: "xxx.xx.x/eccs-vault-dockerimages"
tag: "vault_1.9.0_0.18.0"

# The default values for the injected Vault Agent containers.
agentDefaults:
# For more information on configuring resources, see the K8s documentation:
# https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
cpuLimit: "500m"
cpuRequest: "250m"
memLimit: "128Mi"
memRequest: "64Mi"

# Default template type for secrets when no custom template is specified.
# Possible values include: "json" and "map".
template: "map"

# Default values within Agent's template_config stanza.
templateConfig:
exitOnRetryFailure: true
staticSecretRenderInterval: ""

# Mount Path of the Vault Kubernetes Auth Method.
authPath: "auth/kubernetes"

# Configures the log verbosity of the injector.
# Supported log levels include: trace, debug, info, warn, error
logLevel: "info"

# Configures the log format of the injector. Supported log formats: "standard", "json".
logFormat: "standard"

# Configures all Vault Agent sidecars to revoke their token when shutting down
revokeOnShutdown: false

# namespaceSelector is the selector for restricting the webhook to only
# specific namespaces.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector
# for more details.
# Example:
# namespaceSelector:
# matchLabels:
# sidecar-injector: enabled
namespaceSelector: {}
# objectSelector is the selector for restricting the webhook to only
# specific labels.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-objectselector
# for more details.
# Example:
# objectSelector:
# matchLabels:
# vault-sidecar-injector: enabled
objectSelector: {}

# Configures failurePolicy of the webhook. The "unspecified" default behaviour deoends on the
# API Version of the WebHook.
# To block pod creation while webhook is unavailable, set the policy to `Fail` below.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy
#
failurePolicy: Ignore

# Extra annotations to attach to the webhook
webhookAnnotations: {}

certs:
# secretName is the name of the secret that has the TLS certificate and
# private key to serve the injector webhook. If this is null, then the
# injector will default to its automatic management mode that will assign
# a service account to the injector to generate its own certificates.
secretName: vault-server-tls

# caBundle is a base64-encoded PEM-encoded certificate bundle for the CA
# that signed the TLS certificate that the webhook serves. This must be set
# if secretName is non-null, unless an external service like cert-manager is
# keeping the caBundle updated.
caBundle: ""

# certName and keyName are the names of the files within the secret for
# the TLS cert and private key, respectively. These have reasonable
# defaults but can be customized if necessary.
certName: vault.crt
keyName: vault.key

resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 256Mi
cpu: 250m

# extraEnvironmentVars is a list of extra environment variables to set in the
# injector deployment.
extraEnvironmentVars: {}
# KUBERNETES_SERVICE_HOST: kubernetes.default.svc

# Affinity Settings for injector pods
# This can either be multi-line string or YAML matching the PodSpec's affinity field.
# Commenting out or setting as empty the affinity variable, will allow
# deployment of multiple replicas to single node services such as Minikube.
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: {{ template "vault.name" . }}-agent-injector
app.kubernetes.io/instance: "{{ .Release.Name }}"
component: webhook
topologyKey: kubernetes.io/hostname

# Toleration Settings for injector pods
# This should be either a multi-line string or YAML matching the Toleration array
# in a PodSpec.
tolerations: []

# nodeSelector labels for server pod assignment, formatted as a multi-line string or YAML map.
# ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
# Example:
# nodeSelector:
# beta.kubernetes.io/arch: amd64
nodeSelector: {}

# Priority class for injector pods
priorityClassName: ""

# Extra annotations to attach to the injector pods
# This can either be YAML or a YAML-formatted multi-line templated string map
# of the annotations to apply to the injector pods
annotations: {}

# Extra labels to attach to the agent-injector
# This should be a YAML map of the labels to apply to the injector
extraLabels: {}

# Should the injector pods run on the host network (useful when using
# an alternate CNI in EKS)
hostNetwork: false

# Injector service specific config
service:
# Extra annotations to attach to the injector service
annotations: {}

# A disruption budget limits the number of pods of a replicated application
# that are down simultaneously from voluntary disruptions
podDisruptionBudget: {}
# podDisruptionBudget:
# maxUnavailable: 1

server:
# If not set to true, Vault server will not be installed. See vault.mode in _helpers.tpl for implementation details
enabled: true

# [Enterprise Only] This value refers to a Kubernetes secret that you have
# created that contains your enterprise license. If you are not using an
# enterprise image or if you plan to introduce the license key via another
# route, then leave secretName blank ("") or set it to null.
# Requires Vault Enterprise 1.8 or later.
enterpriseLicense:
# The name of the Kubernetes secret that holds the enterprise license. The
# secret must be in the same namespace that Vault is installed into.
secretName: ""
# The key within the Kubernetes secret that holds the enterprise license.
secretKey: "license"

# Resource requests, limits, etc. for the server cluster placement. This
# should map directly to the value of the resources field for a PodSpec.
# By default no direct resource request is made.

image:
repository: "xxx.xx.x/eccs-vault-dockerimages"
tag: "vault_1.9.0_0.18.0"
# Overrides the default Image Pull Policy
pullPolicy: IfNotPresent

# Configure the Update Strategy Type for the StatefulSet
# See https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
updateStrategyType: "OnDelete"

# Configure the logging verbosity for the Vault server.
# Supported log levels include: trace, debug, info, warn, error
logLevel: ""

# Configure the logging format for the Vault server.
# Supported log formats include: standard, json
logFormat: ""

resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 256Mi
cpu: 250m

# Ingress allows ingress services to be created to allow external access
# from Kubernetes to access Vault pods.
# If deployment is on OpenShift, the following block is ignored.
# In order to expose the service, use the route section below
ingress:
enabled: false
labels: {}
# traffic: external
annotations: {}
# |
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
# or
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"

# Optionally use ingressClassName instead of deprecated annotation.
# See: https://kubernetes.io/docs/concepts/services-networking/ingress/#deprecated-annotation
ingressClassName: ""

# As of Kubernetes 1.19, all Ingress Paths must have a pathType configured. The default value below should be sufficient in most cases.
# See: https://kubernetes.io/docs/concepts/services-networking/ingress/#path-types for other possible values.
pathType: Prefix

# When HA mode is enabled and K8s service registration is being used,
# configure the ingress to point to the Vault active service.
activeService: true
hosts:
- host: chart-example.local
paths: []
## Extra paths to prepend to the host configuration. This is useful when working with annotation based services.
extraPaths: []
# - path: /*
# backend:
# service:
# name: ssl-redirect
# port:
# number: use-annotation
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local

# OpenShift only - create a route to expose the service
# The created route will be of type passthrough
route:
enabled: false

# When HA mode is enabled and K8s service registration is being used,
# configure the route to point to the Vault active service.
activeService: true

labels: {}
annotations: {}
host: chart-example.local

# authDelegator enables a cluster role binding to be attached to the service
# account. This cluster role binding can be used to setup Kubernetes auth
# method. https://www.vaultproject.io/docs/auth/kubernetes.html
authDelegator:
enabled: true

# extraInitContainers is a list of init containers. Specified as a YAML list.
# This is useful if you need to run a script to provision TLS certificates or
# write out configuration files in a dynamic way.
extraInitContainers: null
# # This example installs a plugin pulled from github into the /usr/local/libexec/vault/oauthapp folder,
# # which is defined in the volumes value.
# - name: oauthapp
# image: "alpine"
# command: [sh, -c]
# args:
# - cd /tmp &&
# wget https://github.com/puppetlabs/vault-plugin-secrets-oauthapp/releases/download/v1.2.0/vault-plugin-secrets-oauthapp-v1.2.0-linux-amd64.tar.xz -O oauthapp.xz &&
# tar -xf oauthapp.xz &&
# mv vault-plugin-secrets-oauthapp-v1.2.0-linux-amd64 /usr/local/libexec/vault/oauthapp &&
# chmod +x /usr/local/libexec/vault/oauthapp
# volumeMounts:
# - name: plugins
# mountPath: /usr/local/libexec/vault

# extraContainers is a list of sidecar containers. Specified as a YAML list.
extraContainers: null

# shareProcessNamespace enables process namespace sharing between Vault and the extraContainers
# This is useful if Vault must be signaled, e.g. to send a SIGHUP for log rotation
shareProcessNamespace: false

# extraArgs is a string containing additional Vault server arguments.
extraArgs: ""

# Used to define custom readinessProbe settings
readinessProbe:
enabled: true
# If you need to use a http path instead of the default exec
# path: /v1/sys/health?standbyok=true

# When a probe fails, Kubernetes will try failureThreshold times before giving up
failureThreshold: 2
# Number of seconds after the container has started before probe initiates
initialDelaySeconds: 5
# How often (in seconds) to perform the probe
periodSeconds: 5
# Minimum consecutive successes for the probe to be considered successful after having failed
successThreshold: 1
# Number of seconds after which the probe times out.
timeoutSeconds: 3
# Used to enable a livenessProbe for the pods
livenessProbe:
enabled: true
path: "/v1/sys/health?standbyok=true"
# When a probe fails, Kubernetes will try failureThreshold times before giving up
failureThreshold: 2
# Number of seconds after the container has started before probe initiates
initialDelaySeconds: 60
# How often (in seconds) to perform the probe
periodSeconds: 5
# Minimum consecutive successes for the probe to be considered successful after having failed
successThreshold: 1
# Number of seconds after which the probe times out.
timeoutSeconds: 3

# Optional duration in seconds the pod needs to terminate gracefully.
# See: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
terminationGracePeriodSeconds: 10

# Used to set the sleep time during the preStop step
preStopSleepSeconds: 5

# Used to define commands to run after the pod is ready.
# This can be used to automate processes such as initialization
# or boostrapping auth methods.
postStart: []
# - /bin/sh
# - -c
# - /vault/userconfig/myscript/run.sh

# extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be
# used to include variables required for auto-unseal.
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
# GOOGLE_REGION: global
# GOOGLE_PROJECT: myproject
# GOOGLE_APPLICATION_CREDENTIALS: /vault/userconfig/myproject/myproject-creds.json

# extraSecretEnvironmentVars is a list of extra environment variables to set with the stateful set.
# These variables take value from existing Secret objects.
extraSecretEnvironmentVars: []
# - envName: AWS_SECRET_ACCESS_KEY
# secretName: vault
# secretKey: AWS_SECRET_ACCESS_KEY

# Deprecated: please use 'volumes' instead.
# extraVolumes is a list of extra volumes to mount. These will be exposed
# to Vault in the path `/vault/userconfig//`. The value below is
# an array of objects, examples are shown below.
extraVolumes:
- type: secret
name: vault-server-tls
# - type: secret (or "configMap")
# name: my-secret
# path: null # default is `/vault/userconfig`

# volumes is a list of volumes made available to all containers. These are rendered
# via toYaml rather than pre-processed like the extraVolumes value.
# The purpose is to make it easy to share volumes between containers.
volumes: null
# - name: plugins
# emptyDir: {}

# volumeMounts is a list of volumeMounts for the main server container. These are rendered
# via toYaml rather than pre-processed like the extraVolumes value.
# The purpose is to make it easy to share volumes between containers.
volumeMounts: null
# - mountPath: /usr/local/libexec/vault
# name: plugins
# readOnly: true

# Affinity Settings
# Commenting out or setting as empty the affinity variable, will allow
# deployment to single node services such as Minikube
# This should be either a multi-line string or YAML matching the PodSpec's affinity field.
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: {{ template "vault.name" . }}
app.kubernetes.io/instance: "{{ .Release.Name }}"
component: server
topologyKey: kubernetes.io/hostname

# Toleration Settings for server pods
# This should be either a multi-line string or YAML matching the Toleration array
# in a PodSpec.
tolerations: []

# nodeSelector labels for server pod assignment, formatted as a multi-line string or YAML map.
# ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
# Example:
# nodeSelector:
# beta.kubernetes.io/arch: amd64
nodeSelector: {}

# Enables network policy for server pods
networkPolicy:
enabled: false
egress: []
# egress:
# - to:
# - ipBlock:
# cidr: 10.0.0.0/24
# ports:
# - protocol: TCP
# port: 443

# Priority class for server pods
priorityClassName: ""

# Extra labels to attach to the server pods
# This should be a YAML map of the labels to apply to the server pods
extraLabels: {}

# Extra annotations to attach to the server pods
# This can either be YAML or a YAML-formatted multi-line templated string map
# of the annotations to apply to the server pods
annotations: {}

# Enables a headless service to be used by the Vault Statefulset
service:
enabled: true
# clusterIP controls whether a Cluster IP address is attached to the
# Vault service within Kubernetes. By default the Vault service will
# be given a Cluster IP address, set to None to disable. When disabled
# Kubernetes will create a "headless" service. Headless services can be
# used to communicate with pods directly through DNS instead of a round robin
# load balancer.
# clusterIP: None

# Configures the service type for the main Vault service. Can be ClusterIP
# or NodePort.
#type: ClusterIP

# The externalTrafficPolicy can be set to either Cluster or Local
# and is only valid for LoadBalancer and NodePort service types.
# The default value is Cluster.
# ref: https://kubernetes.io/docs/concepts/services-networking/service/#external-traffic-policy
externalTrafficPolicy: Cluster

# If type is set to "NodePort", a specific nodePort value can be configured,
# will be random if left blank.
#nodePort: 30000

# Port on which Vault server is listening
port: 8200
# Target port to which the service should be mapped to
targetPort: 8200
# Extra annotations for the service definition. This can either be YAML or a
# YAML-formatted multi-line templated string map of the annotations to apply
# to the service.
annotations: {}

#loadBalancerIP: {}
#port: 30075
#type: LoadBalancer

# This configures the Vault Statefulset to create a PVC for data
# storage when using the file or raft backend storage engines.
# See https://www.vaultproject.io/docs/configuration/storage/index.html to know more
dataStorage:
enabled: true
# Size of the PVC created
size: 10Gi
# Location where the PVC will be mounted.
mountPath: "/vault/data"
# Name of the storage class to use. If null it will use the
# configured default Storage Class.
storageClass: eccs-vault-sc
# Access Mode of the storage device being used for the PVC
accessMode: ReadWriteOnce
# Annotations to apply to the PVC
annotations: {}

# This configures the Vault Statefulset to create a PVC for audit
# logs. Once Vault is deployed, initialized and unseal, Vault must
# be configured to use this for audit logs. This will be mounted to
# /vault/audit
# See https://www.vaultproject.io/docs/audit/index.html to know more
auditStorage:
enabled: false
# Size of the PVC created
size: 10Gi
# Location where the PVC will be mounted.
mountPath: "/vault/audit"
# Name of the storage class to use. If null it will use the
# configured default Storage Class.
storageClass: eccs-vault-sc
# Access Mode of the storage device being used for the PVC
accessMode: ReadWriteOnce
# Annotations to apply to the PVC
annotations: {}

# Run Vault in "dev" mode. This requires no further setup, no state management,
# and no initialization. This is useful for experimenting with Vault without
# needing to unseal, store keys, et. al. All data is lost on restart - do not
# use dev mode for anything other than experimenting.
# See https://www.vaultproject.io/docs/concepts/dev-server.html to know more
dev:
enabled: false

# Set VAULT_DEV_ROOT_TOKEN_ID value
devRootToken: "root"

# Run Vault in "standalone" mode. This is the default mode that will deploy if
# no arguments are given to helm. This requires a PVC for data storage to use
# the "file" backend. This mode is not highly available and should not be scaled
# past a single replica.
standalone:
enabled: "-"

# config is a raw string of default configuration when using a Stateful
# deployment. Default is to use a PersistentVolumeClaim mounted at /vault/data
# and store data there. This is only used when using a Replica count of 1, and
# using a stateful set. This should be HCL.

# Note: Configuration files are stored in ConfigMaps so sensitive data
# such as passwords should be either mounted through extraSecretEnvironmentVars
# or through a Kube secret. For more information see:
# https://www.vaultproject.io/docs/platform/k8s/helm/run#protecting-sensitive-vault-configurations
config: |
ui = true

listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "file" {
path = "/vault/data"
}

# Example configuration for using auto-unseal, using Google Cloud KMS. The
# GKMS keys must already exist, and the cluster must have a service account
# that is authorized to access GCP KMS.
#seal "gcpckms" {
# project = "vault-helm-dev"
# region = "global"
# key_ring = "vault-helm-unseal-kr"
# crypto_key = "vault-helm-unseal-key"
#}

# Run Vault in "HA" mode. There are no storage requirements unless audit log
# persistence is required. In HA mode Vault will configure itself to use Consul
# for its storage backend. The default configuration provided will work the Consul
# Helm project by default. It is possible to manually configure Vault to use a
# different HA backend.
ha:
enabled: true
replicas: 3

# Set the api_addr configuration for Vault HA
# See https://www.vaultproject.io/docs/configuration#api_addr
# If set to null, this will be set to the Pod IP Address
apiAddr: null

# Enables Vault's integrated Raft storage. Unlike the typical HA modes where
# Vault's persistence is external (such as Consul), enabling Raft mode will create
# persistent volumes for Vault to store data according to the configuration under server.dataStorage.
# The Vault cluster will coordinate leader elections and failovers internally.
raft:

# Enables Raft integrated storage
enabled: false
# Set the Node Raft ID to the name of the pod
setNodeId: false

# Note: Configuration files are stored in ConfigMaps so sensitive data
# such as passwords should be either mounted through extraSecretEnvironmentVars
# or through a Kube secret. For more information see:
# https://www.vaultproject.io/docs/platform/k8s/helm/run#protecting-sensitive-vault-configurations
config: |
ui = true

listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
}

storage "raft" {
path = "/vault/data"
}

service_registration "kubernetes" {}

# config is a raw string of default configuration when using a Stateful
# deployment. Default is to use a Consul for its HA storage backend.
# This should be HCL.

# Note: Configuration files are stored in ConfigMaps so sensitive data
# such as passwords should be either mounted through extraSecretEnvironmentVars
# or through a Kube secret. For more information see:
# https://www.vaultproject.io/docs/platform/k8s/helm/run#protecting-sensitive-vault-configurations
config: |
ui = true

listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
}
storage "consul" {
path = "vault"
address = "HOST_IP:8500"
}

telemetry {
disable_hostname = true
prometheus_retention_time = "12h"
}

service_registration "kubernetes" {}

# Example configuration for using auto-unseal, using Google Cloud KMS. The
# GKMS keys must already exist, and the cluster must have a service account
# that is authorized to access GCP KMS.
#seal "gcpckms" {
# project = "vault-helm-dev-246514"
# region = "global"
# key_ring = "vault-helm-unseal-kr"
# crypto_key = "vault-helm-unseal-key"
#}

# A disruption budget limits the number of pods of a replicated application
# that are down simultaneously from voluntary disruptions
disruptionBudget:
enabled: true

# maxUnavailable will default to (n/2)-1 where n is the number of
# replicas. If you'd like a custom value, you can specify an override here.
maxUnavailable: null

# Definition of the serviceAccount used to run Vault.
# These options are also used when using an external Vault server to validate
# Kubernetes tokens.
serviceAccount:
# Specifies whether a service account should be created
create: true
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
# Extra annotations for the serviceAccount definition. This can either be
# YAML or a YAML-formatted multi-line templated string map of the
# annotations to apply to the serviceAccount.
annotations: {}

# Settings for the statefulSet used to run Vault.
statefulSet:
# Extra annotations for the statefulSet. This can either be YAML or a
# YAML-formatted multi-line templated string map of the annotations to apply
# to the statefulSet.
annotations: {}

# Vault UI
ui:
# True if you want to create a Service entry for the Vault UI.
#
# serviceType can be used to control the type of service created. For
# example, setting this to "LoadBalancer" will create an external load
# balancer (for supported K8S installations) to access the UI.
enabled: true
publishNotReadyAddresses: true
# The service should only contain selectors for active Vault pod
activeVaultPodOnly: true
serviceType: "LoadBalancer"
serviceNodePort: null
externalPort: 32376
targetPort: 8200

# The externalTrafficPolicy can be set to either Cluster or Local
# and is only valid for LoadBalancer and NodePort service types.
# The default value is Cluster.
# ref: https://kubernetes.io/docs/concepts/services-networking/service/#external-traffic-policy
externalTrafficPolicy: Cluster

#loadBalancerSourceRanges:
# - 10.0.0.0/16
# - 1.78.23.3/32

# loadBalancerIP:

# Extra annotations to attach to the ui service
# This can either be YAML or a YAML-formatted multi-line templated string map
# of the annotations to apply to the ui service
annotations: {}

# secrets-store-csi-driver-provider-vault
csi:
# True if you want to install a secrets-store-csi-driver-provider-vault daemonset.
#
# Requires installing the secrets-store-csi-driver separately, see:
# https://github.com/kubernetes-sigs/secrets-store-csi-driver#install-the-secrets-store-csi-driver
#
# With the driver and provider installed, you can mount Vault secrets into volumes
# similar to the Vault Agent injector, and you can also sync those secrets into
# Kubernetes secrets.
enabled: false

image:
repository: "xxx.xx.x/eccs-vault-dockerimages"
tag: "vault-csi-provider_0.3.0_0.18.0"
pullPolicy: IfNotPresent

# volumes is a list of volumes made available to all containers. These are rendered
# via toYaml rather than pre-processed like the extraVolumes value.
# The purpose is to make it easy to share volumes between containers.
volumes: null
# - name: tls
# secret:
# secretName: vault-tls

# volumeMounts is a list of volumeMounts for the main server container. These are rendered
# via toYaml rather than pre-processed like the extraVolumes value.
# The purpose is to make it easy to share volumes between containers.
volumeMounts: null
# - name: tls
# mountPath: "/vault/tls"
# readOnly: true

# resources: {}
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
cpu: 50m
memory: 128Mi

# Settings for the daemonSet used to run the provider.
daemonSet:
updateStrategy:
type: RollingUpdate
maxUnavailable: ""
# Extra annotations for the daemonSet. This can either be YAML or a
# YAML-formatted multi-line templated string map of the annotations to apply
# to the daemonSet.
annotations: {}
# Provider host path (must match the CSI provider's path)
providersDir: "/etc/kubernetes/secrets-store-csi-providers"
# Kubelet host path
kubeletRootDir: "/var/lib/kubelet"

pod:
# Extra annotations for the provider pods. This can either be YAML or a
# YAML-formatted multi-line templated string map of the annotations to apply
# to the pod.
annotations: {}

# Toleration Settings for provider pods
# This should be either a multi-line string or YAML matching the Toleration array
# in a PodSpec.
tolerations: []

serviceAccount:
# Extra annotations for the serviceAccount definition. This can either be
# YAML or a YAML-formatted multi-line templated string map of the
# annotations to apply to the serviceAccount.
annotations: {}

# Used to configure readinessProbe for the pods.
readinessProbe:
# When a probe fails, Kubernetes will try failureThreshold times before giving up
failureThreshold: 2
# Number of seconds after the container has started before probe initiates
initialDelaySeconds: 5
# How often (in seconds) to perform the probe
periodSeconds: 5
# Minimum consecutive successes for the probe to be considered successful after having failed
successThreshold: 1
# Number of seconds after which the probe times out.
timeoutSeconds: 3
# Used to configure livenessProbe for the pods.
livenessProbe:
# When a probe fails, Kubernetes will try failureThreshold times before giving up
failureThreshold: 2
# Number of seconds after the container has started before probe initiates
initialDelaySeconds: 5
# How often (in seconds) to perform the probe
periodSeconds: 5
# Minimum consecutive successes for the probe to be considered successful after having failed
successThreshold: 1
# Number of seconds after which the probe times out.
timeoutSeconds: 3

# Enables debug logging.
debug: false

# Pass arbitrary additional arguments to vault-csi-provider.
extraArgs: []

external_certs:
enabled: true
name: vault-server-tls
data:
vault_ca: ""
vault_crt: ""
vault_key: ""

Start with the injector logs, API-server logs, the chart values, and the nginx Deployment manifest, then inspect how the listed Vault annotations are processed. Reproduce the deployment on Kubernetes 1.21.5 with vault-helm 0.18.0 and verify that the nginx pod is mutated to contain both the nginx and vault-agent containers.

Written by the indexing model from the issue text.