fluent / fluent/fluent-operator

bug: fluent-operator breaks if a secret in a namespace is not available for the output

Open
#1,728 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
682
Forks
328
Avg merge
2d 23h
Merged PRs (30d)
13

Description

### Describe the issue

Hello,

We have outputs in multiple namespaces, and in one of those namespaces, the secret defined in the output wasn't available. This has caused fluent-bit daemon sets to go into CrashLoopBackOff and to not properly start.

There were multiple errors in the operator log where it cannot find the secrets in all namespaces, even though the secrets were present in all of them, except one e.g. test-01:

```
2025-10-14T15:09:27Z  ERROR  Reconciler error  {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "FluentBit": {"name":"fluent-bit-config","namespace":"test-01"}, "namespace": test-01", "name": "fluent-bit-config", "reconcileID": "516abca7-9351-4f88-8b30-ff54fadf9c16", "error": "Secret \"elasticsearch-credentials\" not found"}
2025-10-14T15:09:27Z  ERROR  Reconciler error  {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "FluentBit": {"name":"fluent-bit-config","namespace":"test-02"}, "namespace": "test-02", "name": "fluent-bit-config", "reconcileID": "4feafd59-972e-444d-a861-b36185091492", "error": "Secret \"elasticsearch-credentials\" not found"}
2025-10-14T15:09:27Z  ERROR  Reconciler error  {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "FluentBit": {"name":"fluent-bit-config","namespace":"test-03"}, "namespace": "test-03", "name": "fluent-bit-config", "reconcileID": "850d08bf-a5d8-4109-afc6-1bd6dcd2bf07", "error": "Secret \"elasticsearch-credentials\" not found"}
```

Re-creating the missing secret in the namespace test-01 fixed the issue.

Thanks,
Marjan

### To Reproduce

- Create one output per namespace (e.g. more than 3)
- Create the secret in each namespace that is used by the output
- Watch logs being collected
- Delete a secret from one of the namespaces
- Check the logs in the fluent-operator
- Restart one of the fluent-bit daemon sets

### Expected behavior

When a secret for the output is not available, I expect the fluent-operator to report it as an error, but not to crash the fluent-bit instances.

### Your Environment

```markdown
- Fluent Operator version: 3.4.0
- Container Runtime: containerd
- Operating system:
- Kernel version:
```

### How did you install fluent operator?

Via Helm.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue by creating outputs and secrets across namespaces, deleting one secret, and restarting a Fluent Bit DaemonSet while watching fluent-operator logs. Trace the FluentBit reconciliation path that reports the missing secret; done means the operator reports the missing secret as an error without causing Fluent Bit instances to crash.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.