crossplane / crossplane/upjet

State file generated before running first plan/apply contains information about resource from Crossplane

Open
#520 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
481
Forks
131
Avg merge
2d 1h
Merged PRs (30d)
11

Description

### What happened?

I am working on creating a Crossplane provider for [FluxCD](https://fluxcd.io/). The repo for this project is here: https://github.com/alanrichman/provider-flux

I have been using the Upjet docs to create a Crossplane resource that represents the [`flux_bootstrap_git`](https://registry.terraform.io/providers/fluxcd/flux/latest/docs/resources/bootstrap_git) Terraform resource. I have it to the point where it generates Terraform config and is able to apply it successfully if I supply every field the resource has. I am running into a problem when I try to rely on default values that the provider sets. Take this example manifest for the resource in Crossplane:

```yaml
apiVersion: bootstrap.flux.crossplane.io/v1alpha1
kind: Git
metadata:
name: test
spec:
forProvider:
path: "clusters/provider-flux"
version: "v2.6.4"
namespace: "flux-system"
# secretName: "flux-system"
components:
- "source-controller"
- "kustomize-controller"
- "helm-controller"
- "notification-controller"
componentsExtra:
- image-reflector-controller
- image-automation-controller
watchAllNamespaces: true
networkPolicy: true
logLevel: "info"
interval: "1m0s"
deleteGitManifests: true
providerConfigRef:
name: default
```

This contains every piece of information that the Terraform provider needs to provision the resource, except for the commented `secretName` field. I can apply this resource to the cluster and the provider (running with `make run`) will create a `main.tf.json` and `terraform.tfstate` file. When the provider attempts to apply it is met with this error:

```
cannot run plan: plan failed: Instance cannot be destroyed: Resource flux_bootstrap_git.test has lifecycle.prevent_destroy set, but the plan calls for this resource to be destroyed. To avoid this error and continue with the plan, either disable lifecycle.prevent_destroy or reduce the scope of the plan using the -target flag.
```

If I manually remove the `prevent_destroy` lifecycle policy and run a plan using the generated config and state I can see that the TF provider is attempting to set the `secret_name` to `flux-system` as is the default from the provider. However the state file has it recorded as `null`. The provider detects that this field forces a delete/recreate but it is not able to do this because of the lifecycle policy. What I am trying to figure out is why the state file is being populated in this way _before_ the `init`/`plan`/`apply` takes place? I have a gist [here](https://gist.github.com/alanrichman/2050103e00e5947748626c6c7dd7392b) that shows the full state file before and after the `init` phase.

If I specify `secretName: flux-system` in the manifest for the `gits.boostrap.flux.crossplane.io` resource the whole thing works as expected with no issues that I can see. Am I missing something about Crossplane providers and Terraform defaults? Attempting to set `r.TerraformResource.Schema["secret_name"].Default = "flux-system"` to reinforce the default of the provider does not appear to make a difference.

### How can we reproduce it?

The most effective way to reproduce this would be to create a Kubernetes cluster somewhere (I use GCP but you could do AWS, local with Kind, it does not matter). Then create a `ProviderConfig` with this manifest

```yaml
apiVersion: flux.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
name: default
namespace: crossplane-system
spec:
credentials:
source: Secret
secretRef:
name: provider-flux
namespace: crossplane-system
key: credentials
```

I am creating a secret in the cluster hosting the CRDs using this template

```json
{
"git_url": "${git_url}",
"git_ssh_username": "${git_ssh_username}",
"git_ssh_private_key": "${git_ssh_private_key}",
"kube_host": "${kube_host}",
"kube_cluster_ca_certificate": "${kube_cluster_ca_certificate}",
"kube_token": "${kube_token}"
}
```

and creating the secret in that cluster with Terraform

```hcl
resource "kubernetes_secret" "provider_flux_credentials" {
metadata {
name = "provider-flux"
namespace = "crossplane-system"
}

data = {
"credentials" = templatefile("./creds.tmpl", {
git_url = "ssh://git@gitlab.com/${data.gitlab_project.applications.path_with_namespace}.git"
git_ssh_username = "git"
git_ssh_private_key = replace(tls_private_key.flux.private_key_pem, "\n", "\\n")

kube_host = "https://${google_container_cluster.primary.endpoint}"
kube_cluster_ca_certificate = replace(base64decode(google_container_cluster.primary.master_auth[0].cluster_ca_certificate), "\n", "\\n")
kube_token = data.google_client_config.default.access_token
})
}
}
```

You can apply this manifest to the cluster hosting the CRDs, then observe the logs from `make run` to see where the TF config is being stored locally

```yaml
apiVersion: bootstrap.flux.crossplane.io/v1alpha1
kind: Git
metadata:
name: test
spec:
forProvider:
path: "clusters/provider-flux"
version: "v2.6.4"
namespace: "flux-system"
# secretName: "flux-system"
components:
- "source-controller"
- "kustomize-controller"
- "helm-controller"
- "notification-controller"
componentsExtra:
- image-reflector-controller
- image-automation-controller
watchAllNamespaces: true
networkPolicy: true
logLevel: "info"
interval: "1m0s"
deleteGitManifests: true
providerConfigRef:
name: default
```

All that to say I am not sure it is strictly necessary to set up the full stack to see the issue. I suspect that as long as you have a workable secret in your local cluster the provider is using that it will generate the TF config correctly.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.