[Feature Request] Consider using explicit configuration for temporal api key

Is your feature request related to a problem? Please describe.

TLDR; it doing weird magic stuff with regards to secrets management and that makes maintenance hard.

I am an application developer and tend to touch k8s the most on our team. Note that I am not the person who set the controller up in our environment. I am aware of it and have had my struggles with it, but its not at the front of my mind in general and I usually am just working with kubernetes abstractions.

In this situation I needed to rotate our temporal key. I look at our application secrets and notice that we have a TEMPORAL_API_KEY in our secrets (which are used via secretRef). I keep updating it checking it, triple checking it, rolling containers, no matter what I do, whenever I exec into the container I see the old api key.

Eventually I noticed we have another kubernetes secret named temporal-api-token that does have the old key in it, but the key is under the api-key value, not TEMPORAL_API_KEY which is what our code uses. I search all our helm charts, our entire codebase for either temporal-api-token or api-key and I find nothing in there. The way this is included in our helm chart it just creates an env var named api-key which everything ignores.

Eventually I track it down. What is happening is that we should not have had a TEMPORAL_API_KEY in secrets at all, that is something that the temporal worker controller creates. It just implicitly assumes that you have a secret named temporal-api-token, and that it has that api-key in it. None of it explicit, its all by convention and it then overwrites the actual environment variables when it starts. This is the only place I can find where any of this stuff is explained and it's frankly really confusing what you're supposed to do and what the options are.

All in all, this was a solid 3 hours of confusion and pulling my hair out over what might be going on.

Describe the solution you'd like

My recommendation for yall:

Document this a hell of a lot more clearly in a lot more places
Error if the TEMPORAL_API_KEY already exists and is going to get overwritten
Explicitly pass in the secret/api-key that will be used into the TemporalWorkerDeployment, This is not a spot for convention-over-configuration. I predict a lot of scenarios where someone sees that secret, finds no reference to it at all in their yaml templates, figures it is some sort of vestigial thing and deletes it then next time the containers restart they fail to come up.

Additional Context

Originally posted in slack

Start with docs/configuration.md and inspect the TemporalWorkerDeployment configuration and the controller's secret and environment-variable handling. Clarify how an explicit secret/api-key reference, collision errors, and documentation should work; done means the proposed behavior and configuration are defined without relying on the current implicit convention.