crossplane / crossplane/upjet

Upon provider pod crash/restart, MR deletion leaves orphaned external resources on cloud provider

Open
#304 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
481
Forks
131
Avg merge
2d 1h
Merged PRs (30d)
11

Description

Resource deletion and orphaned resources on cloud provider

### What happened?

Hello, I created a benchmark to perform tests with thousands of GCP objects. During the MR deletion part of my benchmark, sometimes (less than 1% of managed resources) the MR is deleted from K8s cluster but not the external resource from GCP. The benchmark has been made on the GCP provider, but I think the root cause is in the upjet framework generator. GCP provider in version ***v0.37***, upjet version ***v0.11.0-rc.0***.

\> Here a link to the my [Run N°4: 4000 RecordSets: Create, shutdown then Delete](https://github.com/orange-cloudfoundry/crossplane-benchmark/tree/main/benchmark_gcp_18-10-2023#run-n4-4000-recordsets-create-shutdown-then-delete).
On this run, I have created 4000 RecordSets MRs. Then, wait for the MR object to be in ready state. Scale K8s cluster to 0 worker node, then scale back to 1 worker node. After requesting deletion of all RecordSets some of then are been deleted from Kubernetes, but not from GCP console.
![image](https://github.com/orange-cloudfoundry/crossplane-benchmark/assets/23292338/42433450-a619-4d5b-ac5c-007f6dd4376f)
*Top diagram: Number of API calls reveiced by GCP for the DNS API endpoints. Botton diagram: CPU usage of the worker node*

The CPU consumption of the node is 60%, meaning 10vCPU usage for the provider. During my other benchmark the provider has never been able to consume more than 10vCPU. **This means that it is problably already consuming the maximum amount of available CPU time.**

For example the object *recordset.dns.gcp.upbound.io/benchmarksix10* is missing the Terraform deletion log. This object is still present on GCP console. Object *recordset.dns.gcp.upbound.io/benchmarksix11* is correctly deleted and not present anymore on GCP console.
![image](https://github.com/orange-cloudfoundry/crossplane-benchmark/assets/23292338/86e26452-ea08-48c1-abb5-fb1688d8c785)
Please find the complete log export (exported from GCP Logs Explorer):
- [benchmarksix10.csv](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/benchmarksix10.csv), [benchmarksix10.json](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/benchmarksix10.json)
- [benchmarksix11.csv](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/benchmarksix11.csv), [benchmarksix11.json](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/benchmarksix11.json)

For the object benchmarksix10 this two logs are missing:
```log
2023-10-11T15:49:08Z DEBUG events Successfully requested deletion of external resource [...]
2023-10-11T15:49:08Z DEBUG provider-gcp Successfully requested deletion of external resource [...]
```

\> Here a link to the my [Run N°6: 2000 Buckets](https://github.com/orange-cloudfoundry/crossplane-benchmark/tree/main/benchmark_gcp_18-10-2023#run-n6-2000-buckets). The run deploy 2000 Bucket at (1 rq/s), and then request delete buckets delete. Same behavior as the run n°4: Bucket deleted from K8s but not from GCP

- Correctly deleted in GCP and K8s: [bucketbenchone1929.csv](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/bucketbenchone1929.csv), [bucketbenchone1929.json](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/bucketbenchone1929.json)
- Not deleted in GCP: [bucketbenchone1928.csv](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/bucketbenchone1928.csv), [bucketbenchone1928.json](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/bucketbenchone1928.json)

For the object bucketbenchone1928, this log is present:
```log
2023-10-20T17:30:44Z INFO provider-gcp Failure limit has been exceeded [...]
```

### Expected behavior ?

When requesting deletion of an object in K8s/Crossplane, the provider should wait the deletion confirmation from GCP/Terraform. Here problably the terraform destroy logic have not finish correctly but the object was deleted anyway.

### How can we reproduce it?

The project [benchmark_gcp_18-10-2023](https://github.com/orange-cloudfoundry/crossplane-benchmark/tree/main/benchmark_gcp_18-10-2023#run-n4-4000-recordsets-create-shutdown-then-delete) contains a description of how I build my Crossplane environment (installation, configuration and script)s:
- **Date:** 18/10/2023
- **Platform:** Google Cloud Platform (GCP), GKE Cluster Standard
- **Version:** K8s: 1.27.3-gke.100, Crossplane: v1.13, [provider-gcp-dns](https://marketplace.upbound.io/providers/upbound/provider-gcp-dns/v0.37.0):v0.37.0
- **Setting:** Default, with Crossplane ControllerConfig: *--debug* and *--enable-management-policies*
- **Compute Machine type**: e2-highcpu-16 (16vCPU/16Go)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.