fluent / fluent/fluent-operator
bug: Configuration not being live-reloaded
- Dominant language
- Go
- Stars
- 682
- Forks
- 328
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 13
Description
### Describe the issue
I'm not sure if this is the right repository to open this issue, but the problem seems to be in the live-reloading feature in the `kubesphere/fluent-bit` image. It seems the configuration is not updated until I forcibly restart the pod.
### To Reproduce
We updated the host value in the cluster output:
Before:
```
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
labels:
fluentbit.fluent.io/component: logging
fluentbit.fluent.io/enabled: "true"
name: my-output
spec:
alias: my_pattern
customPlugin:
config: |
Name http
Match my_pattern
Host ingress.coralogix.com
Port 443
URI /logs/v1/singles
Format json_lines
TLS On
Header Authorization Bearer
compress gzip
Retry_Limit 10
match: my_pattern
```
After:
```
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
labels:
fluentbit.fluent.io/component: logging
fluentbit.fluent.io/enabled: "true"
name: myoutput
spec:
alias: mypattern
customPlugin:
config: |
Name http
Match mypattern
Host ingress.cx498-aws-us-west-2.coralogix.com
Port 443
URI /logs/v1/singles
Format json_lines
TLS On
Header Authorization Bearer
compress gzip
Retry_Limit 10
match: mypattern
```
However, we were still observing that in the fluent-bit logs, it was trying to send the logs to:
```
fluent-bit-45lzx fluent-bit [2024/02/05 23:15:00] [error] [output:http:mypattern] ingress.coralogix.com:443, HTTP status=403
```
The config generated by the fluent-operator was correct as well:
```
[Output]
Match mypattern
Alias mypattern
Name http
Match mypattern
Host ingress.cx498-aws-us-west-2.coralogix.com
Port 443
URI /logs/v1/singles
Format json_lines
TLS On
Header Authorization Bearer
compress gzip
Retry_Limit 10
```
This only resolved after I did a `kubectl rollout restart daemonset fluent-bit` and then it began sending via the correct configuration:
```
fluent-bit-45lzx fluent-bit [2024/02/06 00:00:52] [ info] [output:http:mypattern] ingress.cx498-aws-us-west-2.coralogix.com:443, HTTP status=200
```
### Expected behavior
The new configuration should be loaded by the fluent-bit pods without a forced restart.
### Your Environment
```markdown
- Fluent Operator version: docker.io/kubesphere/fluent-operator:v2.7.0
- Fluent Bit version: docker.io/kubesphere/fluent-bit:v2.2.2
- Container Runtime: containerd
- Operating system: Amazon Linux 2023
- Kernel version: 5.10.205-195.804.amzn2.x86_64
```
### How did you install fluent operator?
I'm using the helm chart. This is deployed to AWS EKS.
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.