fluent / fluent/fluent-operator

bug: Configuration not being live-reloaded

Open
#1,058 1 comment 4 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
682
Forks
328
Avg merge
2d 23h
Merged PRs (30d)
13

Description

### Describe the issue

I'm not sure if this is the right repository to open this issue, but the problem seems to be in the live-reloading feature in the `kubesphere/fluent-bit` image. It seems the configuration is not updated until I forcibly restart the pod.

### To Reproduce

We updated the host value in the cluster output:

Before:
```
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
labels:
fluentbit.fluent.io/component: logging
fluentbit.fluent.io/enabled: "true"
name: my-output
spec:
alias: my_pattern
customPlugin:
config: |
Name http
Match my_pattern
Host ingress.coralogix.com
Port 443
URI /logs/v1/singles
Format json_lines
TLS On
Header Authorization Bearer
compress gzip
Retry_Limit 10
match: my_pattern
```

After:
```
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
labels:
fluentbit.fluent.io/component: logging
fluentbit.fluent.io/enabled: "true"
name: myoutput
spec:
alias: mypattern
customPlugin:
config: |
Name http
Match mypattern
Host ingress.cx498-aws-us-west-2.coralogix.com
Port 443
URI /logs/v1/singles
Format json_lines
TLS On
Header Authorization Bearer
compress gzip
Retry_Limit 10
match: mypattern
```

However, we were still observing that in the fluent-bit logs, it was trying to send the logs to:
```
fluent-bit-45lzx fluent-bit [2024/02/05 23:15:00] [error] [output:http:mypattern] ingress.coralogix.com:443, HTTP status=403
```

The config generated by the fluent-operator was correct as well:
```
[Output]
Match mypattern
Alias mypattern
Name http
Match mypattern
Host ingress.cx498-aws-us-west-2.coralogix.com
Port 443
URI /logs/v1/singles
Format json_lines
TLS On
Header Authorization Bearer
compress gzip
Retry_Limit 10
```

This only resolved after I did a `kubectl rollout restart daemonset fluent-bit` and then it began sending via the correct configuration:

```
fluent-bit-45lzx fluent-bit [2024/02/06 00:00:52] [ info] [output:http:mypattern] ingress.cx498-aws-us-west-2.coralogix.com:443, HTTP status=200
```

### Expected behavior

The new configuration should be loaded by the fluent-bit pods without a forced restart.

### Your Environment

```markdown
- Fluent Operator version: docker.io/kubesphere/fluent-operator:v2.7.0
- Fluent Bit version: docker.io/kubesphere/fluent-bit:v2.2.2
- Container Runtime: containerd
- Operating system: Amazon Linux 2023
- Kernel version: 5.10.205-195.804.amzn2.x86_64
```

### How did you install fluent operator?

I'm using the helm chart. This is deployed to AWS EKS.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.