Concierge credential exchange fails because of an outdated ID token

What happened?

I have a local Kind cluster setup running Concierge v0.10.0 configured with Dex v2.30.0. When I have a clean install it works okay, but once I recreate that same setup I get an authentication failure. By recreating I mean deleting the old cluster, creating a new one with Concierge and Dex redeployed, and also regenerating the CA bundle used by Dex.

I’m able to get the config file using the Pinniped CLI without issues, but it seems that there's a problem with verifying the ID Token I get from Dex.

The errors I’m seeing are -

kubectl --kubeconfig my-cluster.yaml get pods -n auth-with-dex
Error: could not complete Concierge credential exchange: login failed: authentication failed
Unable to connect to the server: getting credentials: exec: executable /usr/local/bin/pinniped failed with exit code 1

and the logs from the Concierge pod are showing -

I0810 16:51:07.955524       1 trace.go:205] Trace[1124895541]: "create" kind: (10-Aug-2021 16:51:07.946) (total time: 8ms):
Trace[1124895541]: ---"failure" failureType:token authentication,msg:oidc: verify token: failed to verify signature: failed to verify id token signature 8ms (16:51:00.955)
Trace[1124895541]: [8.718934ms] [8.718934ms] END

What did you expect to happen?

I should have been redirected to Dex where I can log in.

What is the simplest way to reproduce this behavior?

Recreate an environment without changing any parameters and clearing the pinniped cache folder.

In what environment did you see this bug?

• Pinniped server version: v0.10.0
• Pinniped client version: v0.10.0
• Pinniped container image (if using a public container image): projects.registry.vmware.com/pinniped/pinniped-server:v0.10.0@sha256:3bdfb9ad9275449f07614081eca27cff16f15562aeabfee9214a0b0506bb6320
• Pinniped configuration (what IDP(s) are you using? what downstream credential minting mechanisms are you using?): Dex v2.30.0
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T17:56:19Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-06-22T04:47:11Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"darwin/amd64"}

• Kubernetes installer & version (e.g., kubeadm version): kind image version - kindest/node:v1.21.1
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a): darwin 20.6.0
• Others:

What else is there to know about this bug?

Clearing the ~/.config/pinniped folder fixes the issue.

Start with the Concierge credential exchange and Pinniped CLI authentication flow, then inspect how credentials in ~/.config/pinniped are cached and reused after cluster recreation. Reproduce the failure with a recreated Kind cluster and cleared cache, and verify that authentication succeeds without ID token signature errors.