crossplane-contrib / crossplane-contrib/provider-upjet-github
SafeStart watches ALL cluster CRDs at startup → controller-manager cache-sync timeout (2m) / CrashLoopBackOff on large clusters
- Dominant language
- Go
- Stars
- 56
- Forks
- 46
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 2
Description
### **What happened**
On a cluster with many CRDs, provider-upjet-github v0.19.0 never finishes starting: the pod goes into CrashLoopBackOff and the Provider stays HEALTHY=False. The container exits with code 1 (not OOMKilled) ~2 minutes after start.
The failure is intermittent — occasionally it starts and runs for a few minutes, then may crash again on the next restart. Once it manages to start, it works "normally".
**Exact error (the Kind is different on every restart)**
`provider: error: Cannot start controller manager: failed to wait for
managed/team.github.upbound.io/v1alpha1, kind=teammembership caches to sync
kind source: *v1alpha1.TeamMembership: timed out waiting for cache to be synced
for Kind *v1alpha1.TeamMembership`
Across restarts the reported kind rotates randomly: TeamMembership, TeamSettings, RepositoryCustomProperty… — i.e. it isn’t one specific CRD, it’s the aggregate cache sync missing the deadline.
**Root cause (with debug enabled)**
With PROVIDER_DEBUG=true, the startup log emits gvk is ready for ~896 GVKs — essentially every CRD in the cluster, not just the provider’s own GitHub types:
Group count
azure.upbound.io 319
azure.m.upbound.io 239
azuread.* 70
github.* 72
non-crossplane CRDs ~100
total ~896 (cluster has ~801 CRDs)
This comes from the SafeStart path in cmd/provider/main.go:
customresourcesgate.Setup (crossplane-runtime) watches **every CustomResourceDefinition** in the cluster — that’s the source of the ~896 gvk is ready lines. A GitHub provider has no reason to watch Azure/Velero/Traefik CRDs, and its startup cost ends up scaling with the total cluster CRD count, not its own ~78 types.
Combined with the fact that the manager is created without a CacheSyncTimeout:
it inherits controller-runtime’s default CacheSyncTimeout = 2 * time.Minute. On a CRD-heavy multi-provider cluster the SafeStart gate + informers can’t all sync within 2 minutes, so mgr.Start() aborts and the process exits.
There is also no flag to disable SafeStart
**Ruled out (so it isn’t misattributed)**
CPU/memory: crashes with requests.cpu: 1; cpu: 4 is unschedulable on our nodes — not resource-bound, not OOM.
Client-side REST throttling: set --max-reconcile-rate=2000 (20× default) → no effect, still crashes at exactly ~2 min.
Slow API server: individual LISTs of the failing kinds return <1s.
Broken aggregated APIServices: 0 unavailable.
No workaround flag: --help/--help-long expose only --debug, --sync, --poll, --poll-state-metric, --leader-election, --max-reconcile-rate, --enable-management-policies — no cache-sync-timeout, no SafeStart toggle.
Environment
Provider: xpkg.upbound.io/crossplane-contrib/provider-upjet-github:v0.19.0
Crossplane v2 (namespaced .m. MRs), --enable-management-policies = true
Cluster: AKS, ~801 CRDs total (large, mostly provider-azure-family)
**Expected behavior**
The provider should start its controllers only for its own API groups, and start reliably regardless of how many unrelated CRDs exist in the cluster.
**Requested fixes**
- Scope the SafeStart CRD gate to the provider’s own GVKs instead of watching all cluster CRDs.
- Expose CacheSyncTimeout as a flag/env and set it on the manager ctrl.Options.
- Add a flag/env to disable SafeStart
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in cmd/provider/main.go and trace the SafeStart path through customresourcesgate.Setup and manager creation. Verify how the provider’s own GVKs can be selected and where ctrl.Options receives cache settings; done means unrelated CRDs no longer block startup, the requested timeout and SafeStart controls are available, and the provider starts reliably on CRD-heavy clusters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100