fluent / fluent/fluent-operator

help request: Why are cluster-wide permission required for some resources such as DaemonSets, Secrets & more?

Open
#1,281 8 comments 3 reactions 0 assignees View on GitHub
controller scope good first issue help wanted
Dominant language
Go
Stars
682
Forks
328
Avg merge
2d 23h
Merged PRs (30d)
13

Description

### Describe the issue

I am looking to rollout Fluent Operator in an enterprise Kubernetes cluster to handle various telemetry needs for 100+ services. The configuration related CRDs look to allow our service owners to extend our telemetry pipeline for their own needs, such as additional outputs. That's the main reason Fluent Operator stood out for me.

Security is a concern with full cluster-wide permissions given on DaemonSets, StatefulSets, Secrets, ServiceAccounts, and more. Given that these are namespace scoped what functionality does the operator provide that requires cluster-wide permissions on these resources?

I've attempted to reduce the permission scope on these resources to a single namespace using Roles/RoleBindings, but the operator attempts to list/watch these resources across the entire cluster.

For example, I moved permissions below from the ClusterRole to a namespaced Role (with the appropriate bindings). If the operator exists in a single namespace, as well as either FluentBit or Fluentd, then I would not expect that it needs to monitor the entire cluster for these resources.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: telemetry
name: fluent-operator
rules:
- apiGroups:
- apps
resources:
- daemonsets
- statefulsets
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- rbac.authorization.k8s.io
resources:
- clusterrolebindings
verbs:
- create
- list
- get
- watch
- patch
- apiGroups:
- rbac.authorization.k8s.io
resources:
- clusterroles
verbs:
- create
- list
- get
- watch
- patch
- apiGroups:
- ""
resources:
- secrets
- configmaps
- services
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
```

The logs below are from the operator once those permissions were scoped to a single namespace.
```
fluent-operator W0802 20:15:54.123352 1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:telemetry:fluent-operator" cannot list resource "daemonsets" in API group "apps" at the cluster scope
fluent-operator E0802 20:15:54.123404 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: Failed to watch *v1.DaemonSet: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:telemetry:fluent-operator" cannot list resource "daemonsets" in API group "apps" at the cluster scope
fluent-operator W0802 20:16:07.306561 1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:telemetry:fluent-operator" cannot list resource "services" in API group "" at the cluster scope
fluent-operator E0802 20:16:07.306618 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:telemetry:fluent-operator" cannot list resource "services" in API group "" at the cluster scope
fluent-operator W0802 20:16:10.656780 1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:telemetry:fluent-operator" cannot list resource "secrets" in API group "" at the cluster scope
fluent-operator E0802 20:16:10.656875 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:telemetry:fluent-operator" cannot list resource "secrets" in API group "" at the cluster scope
fluent-operator W0802 20:16:19.079865 1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:telemetry:fluent-operator" cannot list resource "statefulsets" in API group "apps" at the cluster scope
fluent-operator E0802 20:16:19.079921 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:telemetry:fluent-operator" cannot list resource "statefulsets" in API group "apps" at the cluster scope
fluent-operator W0802 20:16:23.153158 1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:telemetry:fluent-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope
fluent-operator E0802 20:16:23.153227 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:telemetry:fluent-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope
```

Is it possible to configure RBAC for Fluent Operator such that it doesn't need cluster-wide permissions on these resources while continuing to allow telemetry to be collected from all namespaces? If not, what functionality does the operator provide that requires such permissions?

### How did you install fluent operator?

```bash
helm upgrade --install fluent-operator --create-namespace -n telemetry
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.