Workbench Databricks Secrets

This refers to the Azure (AWS too? idk) Databricks credentials used for sessions in Workbench. The current state of this is extremely messy and does not support updates to this secret when it expires.

Steps are listed here on how to change this, which shows where all the secrets and whatnot are: https://positpbc.atlassian.net/wiki/spaces/PTD/pages/1546616976/General+Maintenance+and+KTLO#Updating-Azure-Databricks-App-Registration-Client-Secret-in-Workbench

In our code, it appears this is controlled by the following functions:
https://github.com/rstudio/ptd/blob/9889768a0d9ca7356d3682a2849f42d9928174df/team-operator/internal/controller/core/workbench.go#L49
https://github.com/rstudio/ptd/blob/9889768a0d9ca7356d3682a2849f42d9928174df/team-operator/internal/controller/core/workbench.go#L549

There also seems to be a SecretProviderClass being created, along with the corresponding volume mounts in the pod definitions for workbench. This is creating a pod mount as well as a k8s secret. As far as I can tell, neither of these are used. The mounted secret does not reflect a change as it should if the secrets provider was actually managing this. It's also not referencing the entire key string including the clientId, which in theory shouldn't work using the json parsing "jmesPath" feature, I don't see a wildcard being used. We need to find out if this is actually required and remove it if not, OR alternatively use this approach only instead of the built in functions to parse this field. If we let the secrets provider manage this mount, it would update automatically as it should be.

For example:

Name:         databricks-workbench-secrets
Kind:         SecretProviderClass
Spec:
  Parameters:
    Objects:  - jmesPath:
  - objectAlias: client-secret
    path: '"dev-client-secret"'
  objectName: example-staging-databricks.posit.team
  objectType: secretsmanager
  objectVersionLabel: AWSCURRENT

and in the pod definition:

volumeMounts:
         - mountPath: /mnt/secure-config/rstudio/                                                                                                                                                                                                                           
           name: secret-config                                                                                                                                                                                                                                              
           readOnly: true
volumes:
       - name: secret-config                                                                                                                                                                                                                                                
         secret:                                                                                                                                                                                                                                                            
           defaultMode: 420                                                                                                                                                                                                                                                 
           secretName: databricks-workbench-config                                                                                                                                                                                                                          
       - csi:                                                                                                                                                                                                                                                               
           driver: secrets-store.csi.k8s.io                                                                                                                                                                                                                                 
           readOnly: true                                                                                                                                                                                                                                                   
           volumeAttributes:                                                                                                                                                                                                                                                
             secretProviderClass: databricks-workbench-secrets

Side note:
The error "the secret already exists; we do not currently support updating values" was found in the team-operator when this issue was occuring, but may or may not be related. We should double check if it is and rule it out if not. https://github.com/rstudio/ptd/blob/5fe8c8eff224609c1cf157a684b053f61b86cc0a/team-operator/internal/secret.go#L18-L19

Start with team-operator/internal/controller/core/workbench.go around the referenced functions and trace the SecretProviderClass, pod mounts, and built-in secret handling. Check team-operator/internal/secret.go around the reported error and determine whether it is related. Done means the update path is established, unused or duplicate handling is resolved, and secret rotation is verified.