async_tfpluginsdk: race condition prevents setting external-name, duplicating resources over and over
- Dominant language
- Go
- Stars
- 481
- Forks
- 131
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 11
Description
### What happened?
We observe that with our tfpluginsdk-using provider, oftentimes the external-name annotation is not set at all. When the Create of the external-resource succeeds, but the annotation is not set, and the resource can be duplicated, Crossplane creates duplicate resources over and over.
After hours of debugging: the issue is most likely a race condition here: https://github.com/crossplane/upjet/blob/c6d5213346ac30cb169e6bcd6ec2edebdca9f184/pkg/controller/external_async_tfpluginsdk.go#L183-L211
The Create for the resource is fired off asnchronously, without ever joining back the goroutine.
So the writeback to k8s of the updated `mg` can happen before or after Create has properly set the annotation.
Once that happens, we eventually see `.status.atProvider` (i think because the opTracker caches it and picks it up on next reconcile), but no annotation, and on the next reconcile interval,
no external-name present, resource is assumed to be needed to Create, piling up duplicates of the resource.
Contributor guide
Assessment
This issue has not been assessed yet.