crossplane / crossplane/crossplane-runtime
Connect() runs before the WasDeleted check, so a rejected ProviderConfigUsage create can block deletion of a managed resource in a terminating namespace
- Dominant language
- Go
- Stars
- 198
- Forks
- 161
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 9
Description
### What happened?
On a Crossplane v2 cluster (v2.3.3, upjet-based Azure providers v2.7.0) we observed a namespace take **hours** to finish terminating. During that time:
- Azure providers issued **825×** HTTP 403 attempting to create `providerconfigusages.azure.m.upbound.io` in the terminating namespace, with `reason: NamespaceTerminating` ("unable to create new content in namespace ... because it is being terminated"). RBAC allowed every one of these — the rejection is namespace lifecycle, not permissions.
- The kube namespace-controller swept the namespace's content (`deletecollection`, all resource types) at a sustained ~31.6/s the entire time.
Both stopped together the moment the namespace finally finished terminating.
### Root cause (traced in `crossplane-runtime`)
1. In `pkg/reconciler/managed/reconciler.go`, `Reconcile()` calls `r.external.Connect(externalCtx, managed)` unconditionally — **before** the `meta.WasDeleted(managed)` branch that leads to `external.Delete()` and `RemoveFinalizer()`. This is true even when `deletionPolicy: Delete` and the managed resource already has a deletion timestamp.
2. Upjet-generated providers' `Connect()` implementations call `ProviderConfigUsageTracker.Track()` (`pkg/resource/providerconfig.go`) first, which does a Get-then-Create-or-Update `Apply` of the resource's `ProviderConfigUsage`.
3. `Track()`'s only swallowed error class is `IsNotAllowed` (`pkg/resource/resource.go`) — an **internal** sentinel produced by its own `AllowUpdateIf` predicate, not `apierrors.IsForbidden`. A genuine 403 from the API server is not swallowed.
4. If the `ProviderConfigUsage` doesn't currently exist — e.g. it was already removed by the namespace controller's `deletecollection` sweep of that GVR — and the namespace is `Terminating`, the `Create` leg gets a 403 (`NamespaceTerminating`). That error propagates out of `Track()` → out of `Connect()`.
5. The reconciler's `Connect()` error handling (`reconciler.go`, right after the `Connect` call) returns early with `Requeue: true` — **before ever reaching the deletion branch**. `external.Delete()` and `RemoveFinalizer()` never run that cycle.
6. Because the managed resource's own finalizer survives, the namespace never fully drains, so the namespace controller re-sweeps every resource type again, which can strip a freshly-recreated `ProviderConfigUsage` before the next reconcile — restarting the race.
The loop only breaks when a `Track()` create happens to land in a timing gap between namespace-controller sweeps, so resolution looks like a race, not a guarantee. We could not establish an upper bound on how long this can take, or whether it can wedge indefinitely under sustained contention.
### How to reproduce
1. Create a namespace with a namespaced (Crossplane v2) managed resource that references a `ProviderConfig`/`ClusterProviderConfig`, using any upjet-based provider.
2. Let it reconcile normally so its `ProviderConfigUsage` exists.
3. Delete the `ProviderConfigUsage` directly (simulating the namespace-controller sweep having already removed it) at roughly the same time as `kubectl delete namespace ` (or delete the namespace first, then delete the PCU before the next reconcile).
4. Observe: the managed resource's `Connect()` fails with a `NamespaceTerminating` 403 while attempting to recreate the `ProviderConfigUsage`; the managed resource's finalizer is not removed while this keeps failing; the namespace stays `Terminating` far longer than expected.
### Expected behavior
Deleting a namespace holding Crossplane managed resources should complete in bounded, predictable time. At minimum:
- `Track()`/`Connect()` should not need to succeed in order for a managed resource already being deleted to proceed to `external.Delete()` and finalizer removal — or the reconciler should check `WasDeleted()` before calling `Connect()` when deletion doesn't require re-establishing a fresh `ProviderConfigUsage`.
- Alternatively, a `NamespaceTerminating` 403 encountered while tracking usage should be treated as ignorable for a resource that is itself already being deleted, since the `ProviderConfigUsage`'s job (protecting the `ProviderConfig` from premature deletion) is moot once the consuming resource is also on its way out.
### Environment
- crossplane-runtime: (pin to the version in use by provider-family-azure / provider-azure-* v2.7.0 — check `go.mod` at that tag)
- Crossplane: v2.3.3
- Providers: `xpkg.crossplane.io/crossplane-contrib/provider-azure-*:v2.7.0` (upjet-based, Azure family v2 line)
- Kubernetes: AKS
### Additional context
This traffic is easy to miss in per-resource audit breakdowns — it's spread thin across ~400 resource types shipped by the Azure provider family CRDs (~543 calls each in our incident), so no single resource type stands out even though the aggregate was ~9.5% of total cluster audit volume that day. Grouping by `@user.username` for the namespace-controller service account surfaces it; grouping by resource type does not.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in pkg/reconciler/managed/reconciler.go by tracing Reconcile through external.Connect, its error handling, and the WasDeleted branch; then read pkg/resource/providerconfig.go and pkg/resource/resource.go for ProviderConfigUsageTracker.Track and its error handling. Reproduce the NamespaceTerminating case described in the issue and add regression coverage showing that deletion reaches external.Delete and finalizer removal without successful usage tracking.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100