crossplane / crossplane/upjet

async_tfpluginsdk: race condition prevents setting external-name, duplicating resources over and over

Open
#632 7 comments 2 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
481
Forks
131
Avg merge
2d 1h
Merged PRs (30d)
11

Description

### What happened?
We observe that with our tfpluginsdk-using provider, oftentimes the external-name annotation is not set at all. When the Create of the external-resource succeeds, but the annotation is not set, and the resource can be duplicated, Crossplane creates duplicate resources over and over.

After hours of debugging: the issue is most likely a race condition here: https://github.com/crossplane/upjet/blob/c6d5213346ac30cb169e6bcd6ec2edebdca9f184/pkg/controller/external_async_tfpluginsdk.go#L183-L211

The Create for the resource is fired off asnchronously, without ever joining back the goroutine.
So the writeback to k8s of the updated `mg` can happen before or after Create has properly set the annotation.

Once that happens, we eventually see `.status.atProvider` (i think because the opTracker caches it and picks it up on next reconcile), but no annotation, and on the next reconcile interval,
no external-name present, resource is assumed to be needed to Create, piling up duplicates of the resource.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.