Upon provider pod crash/restart, MR deletion leaves orphaned external resources on cloud provider
- Dominant language
- Go
- Stars
- 481
- Forks
- 131
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 11
Description
Resource deletion and orphaned resources on cloud provider
### What happened?
Hello, I created a benchmark to perform tests with thousands of GCP objects. During the MR deletion part of my benchmark, sometimes (less than 1% of managed resources) the MR is deleted from K8s cluster but not the external resource from GCP. The benchmark has been made on the GCP provider, but I think the root cause is in the upjet framework generator. GCP provider in version ***v0.37***, upjet version ***v0.11.0-rc.0***.
\> Here a link to the my [Run N°4: 4000 RecordSets: Create, shutdown then Delete](https://github.com/orange-cloudfoundry/crossplane-benchmark/tree/main/benchmark_gcp_18-10-2023#run-n4-4000-recordsets-create-shutdown-then-delete).
On this run, I have created 4000 RecordSets MRs. Then, wait for the MR object to be in ready state. Scale K8s cluster to 0 worker node, then scale back to 1 worker node. After requesting deletion of all RecordSets some of then are been deleted from Kubernetes, but not from GCP console.

*Top diagram: Number of API calls reveiced by GCP for the DNS API endpoints. Botton diagram: CPU usage of the worker node*
The CPU consumption of the node is 60%, meaning 10vCPU usage for the provider. During my other benchmark the provider has never been able to consume more than 10vCPU. **This means that it is problably already consuming the maximum amount of available CPU time.**
For example the object *recordset.dns.gcp.upbound.io/benchmarksix10* is missing the Terraform deletion log. This object is still present on GCP console. Object *recordset.dns.gcp.upbound.io/benchmarksix11* is correctly deleted and not present anymore on GCP console.

Please find the complete log export (exported from GCP Logs Explorer):
- [benchmarksix10.csv](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/benchmarksix10.csv), [benchmarksix10.json](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/benchmarksix10.json)
- [benchmarksix11.csv](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/benchmarksix11.csv), [benchmarksix11.json](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/benchmarksix11.json)
For the object benchmarksix10 this two logs are missing:
```log
2023-10-11T15:49:08Z DEBUG events Successfully requested deletion of external resource [...]
2023-10-11T15:49:08Z DEBUG provider-gcp Successfully requested deletion of external resource [...]
```
\> Here a link to the my [Run N°6: 2000 Buckets](https://github.com/orange-cloudfoundry/crossplane-benchmark/tree/main/benchmark_gcp_18-10-2023#run-n6-2000-buckets). The run deploy 2000 Bucket at (1 rq/s), and then request delete buckets delete. Same behavior as the run n°4: Bucket deleted from K8s but not from GCP
- Correctly deleted in GCP and K8s: [bucketbenchone1929.csv](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/bucketbenchone1929.csv), [bucketbenchone1929.json](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/bucketbenchone1929.json)
- Not deleted in GCP: [bucketbenchone1928.csv](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/bucketbenchone1928.csv), [bucketbenchone1928.json](https://github.com/orange-cloudfoundry/crossplane-benchmark/blob/main/benchmark_gcp_18-10-2023/benchmark/result/bucketbenchone1928.json)
For the object bucketbenchone1928, this log is present:
```log
2023-10-20T17:30:44Z INFO provider-gcp Failure limit has been exceeded [...]
```
### Expected behavior ?
When requesting deletion of an object in K8s/Crossplane, the provider should wait the deletion confirmation from GCP/Terraform. Here problably the terraform destroy logic have not finish correctly but the object was deleted anyway.
### How can we reproduce it?
The project [benchmark_gcp_18-10-2023](https://github.com/orange-cloudfoundry/crossplane-benchmark/tree/main/benchmark_gcp_18-10-2023#run-n4-4000-recordsets-create-shutdown-then-delete) contains a description of how I build my Crossplane environment (installation, configuration and script)s:
- **Date:** 18/10/2023
- **Platform:** Google Cloud Platform (GCP), GKE Cluster Standard
- **Version:** K8s: 1.27.3-gke.100, Crossplane: v1.13, [provider-gcp-dns](https://marketplace.upbound.io/providers/upbound/provider-gcp-dns/v0.37.0):v0.37.0
- **Setting:** Default, with Crossplane ControllerConfig: *--debug* and *--enable-management-policies*
- **Compute Machine type**: e2-highcpu-16 (16vCPU/16Go)
Contributor guide
Assessment
This issue has not been assessed yet.