Azure / Azure/AKS

[Question] Secret sync without dummy pod when using CSI driver for AKV - ALB use Keyvault based Certificate with Gateway API

Open
#5,312 7 comments 0 reactions 1 assignee Claimed by @shashankbarsin View on GitHub
action-required app-gateway-for-containers question
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Describe scenario**

This could be a bug or a feature request potentially

I am trying to setup App Gateway for Containers in my environment,

I am using the gateway API (not Ingress)

i am trying to store the certs in keyvault and reference them using the secrets-store CSI driver with following SecretProvider:

```
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-tls-wildcard-domain-net
spec:
provider: azure
secretObjects: # secretObjects defines the desired state of synced K8s secret objects
- secretName: ingress-tls-csi-wildcard-domain-net
type: kubernetes.io/tls
data:
- objectName: wildcard-domain-net
key: tls.key
- objectName: wildcard-domain-net
key: tls.crt
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: ""
keyvaultName: # the name of the AKV instance
objects: |
array:
- |
objectName: wildcard-domain-net
objectType: secret
tenantId: "" # the tenant ID of the AKV instance
```

and following gateway config

```
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: gateway-01
namespace: default
annotations:
alb.networking.azure.io/alb-id: "/subscriptions//resourceGroups//providers/Microsoft.ServiceNetworking/trafficControllers/"
spec:
gatewayClassName: azure-alb-external
listeners:
- name: http-listener
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: All
- name: https-listener-wildcard-domain
port: 443
protocol: HTTPS
allowedRoutes:
namespaces:
from: All
tls:
mode: Terminate
certificateRefs:
- kind : Secret
group: ""
name: ingress-tls-csi-wildcard-domain-net
addresses:
- type: alb.networking.azure.io/alb-frontend
value:
```

when i attempt to reference the secret name referenced in my SecretProviderClass - i get an error along the lines of:

```
"message":"Secret 'default/ingress-tls-csi-wildcard-domain-net' not found on the cluster.\n\tPlease create the secret on the cluster."
```

However if i spin up a dummy pod mounting the secret via the provider - then it all works....

e.g.

```
kind: Pod
apiVersion: v1
metadata:
name: dummy-secret-pod
spec:
containers:
- name: busybox
image: registry.k8s.io/e2e-test-images/busybox:1.29-4
command:
- "/bin/sleep"
- "10000"
volumeMounts:
- name: ingress-tls-csi-wildcard-domain-net
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: ingress-tls-csi-wildcard-domain-net
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-tls-wildcard-domain-net"
```

however i don't want to require to create a dummy pod like this in the first place - how do i go about configuring the gateway to reference the CSI based secret so that it correctly creates/imports the cert at create time (like the pod ultimately does)

If this feature does not exist today - please kindly consider baking this in :)

**Question**
Your question

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.