envoyproxy / envoyproxy/gateway

extension manager: missing backendResource CRD/RBAC crashes the controller instead of degrading gracefully

Open
#9,808 7 comments 2 reactions 0 assignees View on GitHub
triage
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

## What happened

With `extensionManagers.backendResources` pointing at a custom CRD (e.g. `foo.v1.bar.com.br`), the controller adds a watch for that GVK. If the CRD is absent — or the controller's ServiceAccount lacks `list`/`watch` RBAC on it — the informer's initial LIST fails and the controller never becomes Ready:

```
Failed to watch ... awsbackends.igw.ifood.com.br is forbidden:
User "system:serviceaccount:...:envoy-gateway" cannot list resource "awsbackends" ...
Could not wait for Cache to sync ... timed out waiting for cache to be synced
for Kind *v1.GatewayClass (and every other watched kind)
unable to start provider ... failed to wait for gatewayapi caches to sync ... timed out
leader election lost
```

The manager exits and the pod crash-loops.

## Impact

A missing custom CRD (or a single missing RBAC rule) takes down the **entire** gateway controller for that GatewayClass — not just routes using the custom backend. All routes (custom and non-custom) stop being reconciled/served.

## Expected behavior

Degrade gracefully: log a warning and skip the watch for that backendResource (disabling only the extension manager for that GVK), while continuing to serve everything else. This matches the existing pattern:

```
ServiceImport CRD not found, skipping ServiceImport watch
```

## Suggested fix

Make `backendResources` watch best-effort — e.g. a `watchOnError: continue` (or `optional: true`) option on the extension manager, so a LIST/RBAC failure disables only the extension manager instead of failing `WaitForCacheSync`.

## Willing to contribute

I'd be happy to work on a PR for this fix. The pattern already exists in the codebase for ServiceImport (skip watch on missing CRD), so applying the same to `backendResources` should be a targeted change. Happy to discuss the design — whether it should be a new `optional`/`watchOnError` field, or just always best-effort like ServiceImport.

## Environment

- Envoy Gateway v1.8.0 (official `envoyproxy/gateway:v1.8.0` image)
- `extensionManagers[].backendResources = [{group: foo.bar.com.br, version: v1alpha1, kind: AWSBackend}]`
- `provider.kubernetes.deploy.type: GatewayNamespace`, `watch.type: Namespaces`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the extension manager's backendResources watch setup and the existing ServiceImport watch that skips a missing CRD. Follow how informer LIST errors reach WaitForCacheSync and the provider startup path. Done means a missing CRD or list/watch RBAC failure logs a warning, disables only that backendResource watch, and leaves the controller Ready for other resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.