We should version secrets and configmaps

If you update a secret (Which causes a new deployment rollout because we have the hash of the secret in our deployments) and this secret contains a syntax error, the rollout of the new replicaset fails. However, if now any pods of your old replicaset restart, they also refer to the wrong secret and crash too, and then everything crashes.

Instead, if we updtate a deployment from software-v1 to software-v2 then software-v2 should point to software-v2-secret and software-v1 should point to software-v1-secret. Only after successfully rolling out software-v2 should we delete the software-v1-secret

I discovered this the hard way when I accidentally uploaded the encrypted version of a secret to my test cluster (forgot to sops decrypt) and then things went bad. Helm also didn't allow me to rollback the Secret for some reason no matter what I did, and then both the old and new replicaset crashed.

So there are two issues here:

1. The old replicaset can refer to the new secret which can lead to strange things

2. If you upload a new secret with helm upgrade and you forgot to sops decrypt it, then helm refuses to update the secret even after subsequent deploys that do sops decrypt the secret, and you're stuck with a broken deploy.

3. we can fix, 2. is some bug I can't figure out why it happens.

Related issue: https://github.com/kubernetes/kubernetes/issues/22368

Start by tracing the Helm deployment, Secret, and ConfigMap resources involved in the rollout described here; the payload names no specific files or tests. Define how software-v1 and software-v2 retain separate configuration, when the old Secret is deleted, and how the failed Helm update case should be handled before implementing or testing the change.