State file generated before running first plan/apply contains information about resource from Crossplane
- Dominant language
- Go
- Stars
- 481
- Forks
- 131
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 11
Description
### What happened?
I am working on creating a Crossplane provider for [FluxCD](https://fluxcd.io/). The repo for this project is here: https://github.com/alanrichman/provider-flux
I have been using the Upjet docs to create a Crossplane resource that represents the [`flux_bootstrap_git`](https://registry.terraform.io/providers/fluxcd/flux/latest/docs/resources/bootstrap_git) Terraform resource. I have it to the point where it generates Terraform config and is able to apply it successfully if I supply every field the resource has. I am running into a problem when I try to rely on default values that the provider sets. Take this example manifest for the resource in Crossplane:
```yaml
apiVersion: bootstrap.flux.crossplane.io/v1alpha1
kind: Git
metadata:
name: test
spec:
forProvider:
path: "clusters/provider-flux"
version: "v2.6.4"
namespace: "flux-system"
# secretName: "flux-system"
components:
- "source-controller"
- "kustomize-controller"
- "helm-controller"
- "notification-controller"
componentsExtra:
- image-reflector-controller
- image-automation-controller
watchAllNamespaces: true
networkPolicy: true
logLevel: "info"
interval: "1m0s"
deleteGitManifests: true
providerConfigRef:
name: default
```
This contains every piece of information that the Terraform provider needs to provision the resource, except for the commented `secretName` field. I can apply this resource to the cluster and the provider (running with `make run`) will create a `main.tf.json` and `terraform.tfstate` file. When the provider attempts to apply it is met with this error:
```
cannot run plan: plan failed: Instance cannot be destroyed: Resource flux_bootstrap_git.test has lifecycle.prevent_destroy set, but the plan calls for this resource to be destroyed. To avoid this error and continue with the plan, either disable lifecycle.prevent_destroy or reduce the scope of the plan using the -target flag.
```
If I manually remove the `prevent_destroy` lifecycle policy and run a plan using the generated config and state I can see that the TF provider is attempting to set the `secret_name` to `flux-system` as is the default from the provider. However the state file has it recorded as `null`. The provider detects that this field forces a delete/recreate but it is not able to do this because of the lifecycle policy. What I am trying to figure out is why the state file is being populated in this way _before_ the `init`/`plan`/`apply` takes place? I have a gist [here](https://gist.github.com/alanrichman/2050103e00e5947748626c6c7dd7392b) that shows the full state file before and after the `init` phase.
If I specify `secretName: flux-system` in the manifest for the `gits.boostrap.flux.crossplane.io` resource the whole thing works as expected with no issues that I can see. Am I missing something about Crossplane providers and Terraform defaults? Attempting to set `r.TerraformResource.Schema["secret_name"].Default = "flux-system"` to reinforce the default of the provider does not appear to make a difference.
### How can we reproduce it?
The most effective way to reproduce this would be to create a Kubernetes cluster somewhere (I use GCP but you could do AWS, local with Kind, it does not matter). Then create a `ProviderConfig` with this manifest
```yaml
apiVersion: flux.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
name: default
namespace: crossplane-system
spec:
credentials:
source: Secret
secretRef:
name: provider-flux
namespace: crossplane-system
key: credentials
```
I am creating a secret in the cluster hosting the CRDs using this template
```json
{
"git_url": "${git_url}",
"git_ssh_username": "${git_ssh_username}",
"git_ssh_private_key": "${git_ssh_private_key}",
"kube_host": "${kube_host}",
"kube_cluster_ca_certificate": "${kube_cluster_ca_certificate}",
"kube_token": "${kube_token}"
}
```
and creating the secret in that cluster with Terraform
```hcl
resource "kubernetes_secret" "provider_flux_credentials" {
metadata {
name = "provider-flux"
namespace = "crossplane-system"
}
data = {
"credentials" = templatefile("./creds.tmpl", {
git_url = "ssh://git@gitlab.com/${data.gitlab_project.applications.path_with_namespace}.git"
git_ssh_username = "git"
git_ssh_private_key = replace(tls_private_key.flux.private_key_pem, "\n", "\\n")
kube_host = "https://${google_container_cluster.primary.endpoint}"
kube_cluster_ca_certificate = replace(base64decode(google_container_cluster.primary.master_auth[0].cluster_ca_certificate), "\n", "\\n")
kube_token = data.google_client_config.default.access_token
})
}
}
```
You can apply this manifest to the cluster hosting the CRDs, then observe the logs from `make run` to see where the TF config is being stored locally
```yaml
apiVersion: bootstrap.flux.crossplane.io/v1alpha1
kind: Git
metadata:
name: test
spec:
forProvider:
path: "clusters/provider-flux"
version: "v2.6.4"
namespace: "flux-system"
# secretName: "flux-system"
components:
- "source-controller"
- "kustomize-controller"
- "helm-controller"
- "notification-controller"
componentsExtra:
- image-reflector-controller
- image-automation-controller
watchAllNamespaces: true
networkPolicy: true
logLevel: "info"
interval: "1m0s"
deleteGitManifests: true
providerConfigRef:
name: default
```
All that to say I am not sure it is strictly necessary to set up the full stack to see the issue. I suspect that as long as you have a workable secret in your local cluster the provider is using that it will generate the TF config correctly.
Contributor guide
Assessment
This issue has not been assessed yet.