envoyproxy / envoyproxy/gateway
add config to configure custom RBAC for the Envoy Proxy deployments?
- Vorherrschende Sprache
- Go
- Sterne
- 3k
- Forks
- 864
- Ø Merge
- 2 T. 2 Std.
- Gemergte PRs (30 T.)
- 140
Beschreibung
### Overview
Currently, when we create Envoy Proxy deployments, we create a `ServiceAccount` to go along with them. I would like to propose new config for the `EnvoyProxy` resource that would allow users to set custom rbac for that generated `ServiceAccount`. My initial idea for the config would be more or less something along these lines:
```yaml
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: envoy-proxy-config
namespace: envoy-gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
rbac:
clusterRoleRules:
- apiGroups: ["gateway.networking.k8s.io"]
resources: ["gateways"]
verbs: ["get", "list", "watch"]
- apiGroups: ["mycustomresources.io"]
resources: ["reallycoolresources"]
verbs: ["get", "list", "watch"]
roleRules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch"]
```
This example would tell Envoy Gateway that when creating a new Envoy Proxy deployment to also create a `ClusterRole` and a `Role` (with `ClusterRoleBinding`/`RoleBinding` that tie them to the generated `ServiceAccount`)
### Why is this needed?
We have an extension to Envoy Gateway and we use a go plugin filter that checks for some of our custom resources from the cluster to determine whether or not it has any work to do for incoming requests. We just have a lightweight dockerfile that takes the stock envoy image and includes our `plugin.so` file on the filesystem. That said, I can also see this being nice for anyone who wants to run an ext_authz/ext_proc/etc. filter as a sidecar to the Envoy Proxy deployment that would similarly need it's own custom permissions.
While we use an extension with a custom Envoy image to create that setup, you could also have a similar setup using `EnvoyPatchPolicy` and the envoy deployment patching config in the `EnvoyProxy` resource to do something similar to add your own sidecars to the Envoy Proxy deployments.
### Concerns
Well for starters, this approach is dangerous because giving an application permission to create RBAC resources can lead to the threat of privilege escalation in the cluster.
Since you don't know when/where a `Gateway` will be created, you can't exactly prepare the RBAC ahead of time as a one-time manual apply step since it will need to be in the same namespace as the Envoy Proxy Service Account. The threat of privilege escalation could be mitigated by making this kind of config an opt-in feature that you have to enable and/or making the installation RBAC that would give Envoy Gateway the permission to create resources like `ClusterRole`/etc. something that does not come with the default installations and must be manually created via a copy/paste from the docs to enable the feature.
### Potential workarounds for these use cases?
I was looking into ways to accomplish this without needing to expose the config in Envoy Gateway, but I couldn't come up with any solution that I was totally happy with. We could expose more extension hooks in the infra manager for this, but then that would prevent those `EnvoyPatchPolicy` and/or other custom sidecar use cases from taking advantage of this.
The main issue stems from the fact that Envoy Proxy deployments are created corresponding to `Gateway` resources which could be in different namespaces.
I could come up with was to create a custom controller that watched for `Gateway` creation events, waited for `EnvoyGateway` to create the Envoy Proxy deployment and `ServiceAccount`, and then have that controller patch the `ServiceAccount` with additional permissions. That is a ton of overhead and maintenance to solve this problem outside of Envoy Gateway and I don't like that pattern, plus it just offloads the same risks to another application in the cluster so it isn't any better in that regard.
I could simply require users to apply the necessary RBAC every time they create a `Gateway`, but that would not be a great user experience since forgetting to do so would lead to a crashing deployment of Envoy Proxy.
I could also have any sidecars make gRPC calls to a static service in a single namespace with pre-applied RBAC and ask it for those resources, but the main concern there would be introducing additional request latency. Out of all the possible solutions, this seems the safest but it is also rather heavy-handed with a lot of added overhead/complexity.
### Addtl. notes
I'd love to hear from others on whether you think this is also something that should be solved with additional config in Envoy Gateway or if you think this whole approach is a terrible idea.
If we do want to move ahead with this here, I'm happy to work on it. I thought I'd start by opening an issue to try and get a conversation about whether anyone else thinks they might have similar use cases or ideas about the best way to solve them.
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.