crossplane / crossplane/crossplane-runtime
Race condition in ProviderConfigUsage allows delete a ProviderConfig while managed resources still terminating
- Dominant language
- Go
- Stars
- 198
- Forks
- 161
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 9
Description
### What happened?
When a ProviderConfig is deleted, the `in-use.crossplane.io` finalizer is supposed to block deletion until all managed resources referencing it are **gone**.
However, the finalizer is removed based on the count of ProviderConfigUsage (hereinafter PCU) objects, not actual managed resources.
PCUs have an ownerReference with `blockOwnerDeletion: true` pointing to their managed resource. When a managed resource is deleted, the Kubernetes garbage collector deletes the PCU, often **before** the managed resource's own finalizers have completed and the resource is fully removed.
This causes the PCU count (users) to drop to 0, the reconciler removes the `in-use.crossplane.io` finalizer, and the ProviderConfig is deleted while managed resources are still terminating.
Once the ProviderConfig is gone, surviving managed resources that reconnect (via `Track()`) recreate PCUs pointing to a now-deleted ProviderConfig. These orphaned PCUs persist indefinitely.
The root cause seems to reside in `pkg/reconciler/providerconfig/reconciler.go`; the deletion decision uses `users > 0` (PCU count) as a proxy for "managed resources still exist," but PCU lifetime is shorter than managed resource lifetime due to the garbage collector honoring `blockOwnerDeletion` on the owner (managed resource) deletion, not on the owner's finalizer completion.
### How can we reproduce it?
1. Create a ProviderConfig and several managed resources referencing it (e.g. 3 Databases).
2. Verify the ProviderConfig shows users: 3 and has the `in-use.crossplane.io` finalizer.
3. Delete all managed resources: `kubectl delete database --all`
4. Immediately delete the `ProviderConfig`: `kubectl delete providerconfig `
5. Observe the `ProviderConfig` is deleted even though managed resources still have finalizers pending (their external resources haven't been cleaned up yet).
6. Run `kubectl get providerconfigusage`, you'll see PCUs with timestamps just seconds old, recreated by managed resources that called `Track()` after the `ProviderConfig` was already deleted.
The race is timing-dependent but highly reproducible when external resource deletion takes more than a few seconds (e.g. cloud database teardown). The faster the GC processes the PCU owner reference, the wider the window.
### What environment did it happen in?
Crossplane version: v2.3.3
Original issue: https://github.com/crossplane-contrib/provider-sql/issues/398
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in pkg/reconciler/providerconfig/reconciler.go, where the deletion decision uses the ProviderConfigUsage count, then trace how managed resources call Track(). Reproduce the race with several managed resources and delayed external deletion using the kubectl steps in the issue. Done means a ProviderConfig is not deleted while managed resources are still terminating and recreated ProviderConfigUsage objects do not remain orphaned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100