Provider :system ClusterRole silently under-permissioned: core persists a truncated ProviderRevision.status.objectRefs and never re-derives it

**Type:** Bug
**Area:** package manager (core) + rbac-manager

## Summary

On one cluster, `ProviderRevision.status.objectRefs` for an **active** provider revision recorded **13 of 48** ManagedResourceDefinitions. rbac-manager derives the provider `:system` ClusterRoles from these lists. As a result, every active provider role on the cluster contained 302 of 337 managed-resource kinds. The roles granted **no rules** for `s3.aws.upbound.io`.

Thus the API server denied (403) all requests from the provider ServiceAccount to the API group of its managed resources. No S3 managed resource reconciled for at least 12.5 hours (see [Timeline](#timeline)).

During this time, the ProviderRevision reported `RuntimeHealthy=True` and `RevisionHealthy=True`. The provider pod was `Running` and `Ready` with 0 restarts. rbac-manager logged `Skipped no-op RBAC ClusterRole apply`. It derived the role correctly from the truncated list and found no difference to apply. The only signal was the reflector 403 errors in the provider log.

Routine status reconciles did not repair the list. A **core restart** also did not: core was replaced mid-incident at T+8 h 34 m, and the list stayed at 13 for four more hours. The repair came 72 s after a second core restart. At that time, a no-op annotation was already present on the ProviderRevision (see [Evidence 4](#4-what-did-and-did-not-repair-it)).

## Environment

| | |
|---|---|
| Crossplane | v2.3.4 core + rbac-manager (upgraded from v2.2.0 12 h before onset, see Timeline) |
| Kubernetes | EKS 1.33 (client-go v0.33.0 in provider) |
| Providers | Upbound AWS family v2.3.0: `provider-family-aws`, `provider-aws-s3`, `provider-aws-iam`, `provider-aws-ec2`, `provider-aws-elbv2`, `provider-aws-route53` |
| MRDs / CRDs | 333 / 248 |
| MRD state | legacy `*.aws.upbound.io` `Active`; `.m` variants `Inactive` |
| MRAP | activates 6 legacy `s3.aws.upbound.io` kinds; iam in both variants; ec2/elbv2/route53 `.m` only |

A second cluster has the **identical** Crossplane version and the **identical** provider package digest. The revision name hash is the image digest prefix, thus the package contents are byte-identical. This second cluster is not affected. We used it as the control.

## Timeline

All times are UTC. **T0 = 2026-08-05 01:38Z** is the first observable evidence of the truncation. The rest of this report refers to the offsets in the second column.

| Absolute | Offset | Event |
|---|---|---|
| 2026-07-29 08:48 | T−17 h | The provider provisions S3 resources for a tenant normally. The recorded list was complete at this point |
| 2026-08-04 13:47 | T−12 h | Crossplane core + rbac-manager upgraded v2.2.0 → v2.3.4 (GitOps-driven, new ReplicaSets for both) |
| 2026-08-04 23:34 | T−2 h | Core pod rescheduled onto the new ReplicaSet (node roll, not a deploy change) |
| **2026-08-05 01:38** | **T0** | rbac-manager rewrites the `:system` role of the storage provider → 302 kinds, `s3.aws.upbound.io` gone. First evidence that the recorded list is short. |
| 2026-08-05 ~04:20 | T+2 h 40 m | First composite blocked. This was the earliest tenant that needed a *new* S3 managed resource after T0 |
| 2026-08-05 07:02 | T+5 h 24 m | A tenant resume creates fresh managed resources. They hang with blank conditions |
| 2026-08-05 09:55 | T+8 h 17 m | Incident triage begins (~2.5 h after the first tenant alert) |
| 2026-08-05 10:12 | T+8 h 34 m | **Core pod replaced** (`…-775fb65547-qp79r` → `…-775fb65547-224p9`, same ReplicaSet, thus a pod deletion, not a template change). Core re-reconciles revisions at startup: rbac-manager rewrites the iam `:system` role a minute later, still at 302 kinds. **`objectRefs` remains 13.** |
| 2026-08-05 11:32 | T+9 h 54 m | The `:system` role of the **Inactive** revision of the same provider is deleted → rbac-manager regenerates it **complete** (the own `objectRefs` of that revision is intact at 48) |
| 2026-08-05 12:10 | T+10 h 32 m | Provider pod replaced → Deployment status write → core transitions `RuntimeHealthy`. **Core writes the revision status here and keeps `objectRefs` at 13.** |
| 2026-08-05 12:16 | T+10 h 38 m | The `:system` role of the active revision is recreated → identical 302 |
| 2026-08-05 13:30 | T+11 h 52 m | Role deleted manually → regenerated → identical 302 again |
| 2026-08-05 14:03 | T+12 h 25 m | rbac-manager restarted with `--debug` → logs `Skipped no-op RBAC ClusterRole apply`, zero writes |
| 2026-08-05 14:10 | T+12 h 32 m | Annotation written on the ProviderRevision → no effect for 4.5 min |
| 2026-08-05 14:14:51 | T+12 h 36 m | `rollout restart deploy/crossplane` (core pod `…-766657454d-qd6nm`). The annotation from T+12 h 32 m is still present |
| 2026-08-05 14:16:03 | T+12 h 37 m | Core writes status: `objectRefs` **13 → 48**. rbac-manager applies the `:system` role for all six member revisions in 2 s. |

Total exposure: **12 h 37 m** from the first evidence to the repair. Approximately 10 h of this time was tenant-visible.

## Evidence

### 1. The truncated status, and the arithmetic it produces

```
# affected cluster
$ kubectl get providerrevision -o json | jq '.status.objectRefs | length'
13 # all ManagedResourceDefinition, all *.s3.aws.m.upbound.io, zero *.s3.aws.upbound.io

# control cluster, same revision name / digest
48 # 24 legacy + 24 .m
```

rbac-manager builds the `:system` role of each member provider from the family-wide union of the `objectRefs` of the active revisions:

| active revision | objectRefs |
|---|---|
| iam | 46 |
| **storage** | **13 (expected: 48)** |
| family | 5 |
| ec2 | 202 |
| elbv2 | 14 |
| route53 | 22 |
| **total** | **302 (expected: 337)** |

This total is exactly the kind count that we observed in the generated roles. The 35 missing kinds are the 11 alphabetically last `s3.aws.m.upbound.io` kinds (`bucketpolicies`, `bucketpublicaccessblocks`, `bucketreplicationconfigurations`, `bucketrequestpaymentconfigurations`, `buckets`, `bucketserversideencryptionconfigurations`, `bucketversionings`, `bucketwebsiteconfigurations`, `directorybuckets`, `objectcopies`, `objects`) plus all 24 `s3.aws.upbound.io` kinds. `s3.aws.upbound.io` was also absent from the multi-group `*/finalizers` rule. That rule listed the other ten groups, which included the legacy variant of every other family.

The union is family-wide. Thus **all 7 active provider roles** on the cluster were short in the same way: the family, iam, ec2, elbv2, and route53 roles too, not only the storage role.

### 2. rbac-manager operates correctly

With `--debug`, a freshly started rbac-manager logged:

```
DEBUG crossplane Skipped no-op RBAC ClusterRole apply
{"controller": "rbac-roles/providerrevision.pkg.crossplane.io",
"role-name": "crossplane:provider::system"}
```

rbac-manager computes the desired rules from the recorded list, compares them with the applied rules, and finds them equal. Two rbac-manager restarts produced **zero** writes. The `resourceVersion` of the affected roles did not change. The role was also deleted and regenerated three times (once by an unknown actor in the cluster, twice manually). Each time it came back byte-identical with 302 kinds.

For contrast: the **inactive** revision of the same provider has its own complete 48-entry `objectRefs`. The same rbac-manager binary deleted and regenerated its role minutes earlier. That role came back **complete**, with 24 kinds in each of the two s3 groups.

### 3. Routine status reconciles do not re-derive `objectRefs`

At **T+10 h 32 m**, the provider pod was replaced. kube-controller-manager wrote the status of the provider Deployment. Core observed that write and transitioned the `RuntimeHealthy` condition of the revision in the same second. That is, core wrote the revision status at that moment and kept `objectRefs` at 13. Two more role regenerations followed (T+10 h 38 m and T+11 h 52 m). Both were identical.

`metadata.managedFields` keeps only the latest write per (manager, operation, subresource). Thus the condition `lastTransitionTime` is the only record that remains of that earlier write.

### 4. What did and did not repair it

```
T+8h34m core pod replaced (deletion; same ReplicaSet) -> objectRefs STILL 13, for 4 more hours
T+12h32m kubectl-annotate writes an annotation on the ProviderRevision -> no effect on its own (4.5 min)
T+12h36m kubectl -n crossplane rollout restart deploy/crossplane (annotation still present)
T+12h37m manager=crossplane, subresource=status write -> objectRefs 13 -> 48 (+72 s from restart)
+2 s rbac-manager logs "Applied RBAC ClusterRole" for the :system role of ALL SIX
member revisions (the family-wide union changed for all of them)
```

Thus **a fresh core process alone is not sufficient**. The restart at T+8 h 34 m shows this. That restart did trigger revision reconciles: rbac-manager rewrote a sibling role a minute later. The annotation alone is also not sufficient. Only the combination repaired the list. An annotation-only edit does not increase `metadata.generation`, thus it cannot force a spec-change reconcile. We cannot explain why its presence at startup makes a difference.

We also cannot fully isolate the annotation as the differential. Between the two core restarts, the cluster also saw a provider pod replacement (T+10 h 32 m), two deletions of the generated ClusterRole (T+10 h 38 m and T+11 h 52 m), and an rbac-manager restart (T+12 h 25 m). Each of these events is a possible contributor to the state that made the second startup re-derive the list.

### 5. Downstream effect

```
E reflector.go:200] "Failed to watch" err="failed to list *v1beta1.Bucket:
buckets.s3.aws.upbound.io is forbidden: User \"system:serviceaccount::\"
cannot list resource \"buckets\" in API group \"s3.aws.upbound.io\" at the cluster scope"
```

This error repeated for every S3 kind, approximately 900 KB of log from one pod. The log also showed `cannot update resource "bucketpolicies/status"`. Thus **already-provisioned** resources no longer received status updates but kept their latched `Ready=True` conditions. They look healthy but are unmanaged. Drift correction, lifecycle changes, and deletions are all blocked.

Composite resources that depend on the group hung with blank conditions on their leaf managed resources. In our case, 7 tenant-facing composites hung. They alerted between 5 minutes and 5 hours 35 minutes after their respective creations.

## What we could not determine

- **What truncated `objectRefs`.** The provider provisioned S3 resources normally at T−17 h, thus the list was complete then. The earliest direct evidence of the truncation is the role rewrite at T0. That window contains the v2.2.0 → v2.3.4 upgrade (T−12 h) and a core pod reschedule (T−2 h). But we cannot attribute the truncation to either event, because the control cluster took the same upgrade and is healthy. The MRDs themselves were untouched (`managedFields` unchanged since March). All 48 s3 MRDs existed on the cluster during the outage, correctly owned by the active revision.
- **Why 13, and why an alphabetical `.m`-only prefix.** 13 of 48 is not an obvious page or size boundary. The emitted role was approximately 19 KB, much smaller than the ClusterRole and etcd limits.
- **Why the second core restart re-derived the list and the first did not.** A restart with no other change was effectively tested at T+8 h 34 m. It did **not** repair the list. The annotation was the only deliberate difference at the second restart. But three other cluster-state changes intervened (see Evidence 4). Thus we cannot attribute the repair to a single action.

## Ruled out

- A provider crash or OOM (0 restarts)
- A wrong active revision
- The ClusterRoleBinding subject (correct)
- The ServiceAccount name (correct)
- The CRD `ownerReferences` (correct, active revision)
- MRD or MRAP drift (unchanged for months, all 48 MRDs present)
- GitOps pruning (the generated roles are not managed in git)
- rbac-manager caches or state that a restart clears (two restarts, zero writes)
- Rate limits
- A version-wide regression in 2.3.4 (the control cluster on 2.3.4 regenerates all 337 kinds correctly)
- The package contents (identical digest)
- A cap on the total kind count (the control cluster enumerates all 337)
- Object counts (the affected cluster has *fewer* CRDs than the control)

## Impact

The failure is a silent and total loss of reconciliation for one API group. No Crossplane status field shows it: `RuntimeHealthy=True`, `RevisionHealthy=True`, no event, no log line above debug level. The failure survived a restart of the writing controller (core), two restarts of the reading controller (rbac-manager), two deletions of the generated role, and a provider pod replacement. In our case, it surfaced approximately 2.5 hours after onset as application-level alerts that looked unrelated. The root-cause analysis took most of one day.

## Suggested changes

1. **Re-derive `objectRefs` on each ProviderRevision reconcile**, and set a condition when the recorded list disagrees with the MRDs and CRDs that the revision owns. A re-derivation at startup is not enough: one core restart mid-incident kept the truncated list in place.
2. **Give rbac-manager an MRD/CRD EventSource.** The `rbac-roles/providerrevision` controller watches only `ProviderRevision` and `ClusterRole`. Thus it cannot detect or repair a stale upstream snapshot. It will regenerate an under-permissioned role forever.
3. **Log the number of granted kinds per generated role at info level.** Then a regression of this shape is visible in the rbac-manager output, not only in provider reflector errors.

## Workaround

Annotate the affected ProviderRevision. Then restart crossplane core. Then make sure that `objectRefs` returns to the expected count. In our case, a core restart alone was not sufficient.

For immediate mitigation, create a standalone ClusterRole and binding that grant the dropped group to the provider ServiceAccount. This role survives regeneration. If the role has no `rbac.crossplane.io/system` label and no ownerReferences, rbac-manager does not adopt it.

Start by tracing ProviderRevision status reconciliation in package manager core and the derived :system ClusterRoles in rbac-manager. Reproduce the truncated objectRefs case and compare the recorded list with the complete ManagedResourceDefinition set; done means identifying why re-derivation fails and adding coverage that prevents under-permissioned roles.

Written by the indexing model from the issue text.