fluent / fluent/fluent-operator

bug: S3 output configuration isn't getting passed into fluent-bit

Open
#1,398 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
682
Forks
328
Avg merge
2d 23h
Merged PRs (30d)
13

Description

### Describe the issue

I'm using the fluent-bit operator helm chart and I have 2 outputs enabled, elasticsearch and S3. When fluent-bit pods start (debug logging), I can see es and stderr are configured outputs, but not S3. Unfortunately, the helm chart deployments works fine and I can't find any errors related to the S3 output.

Note: I'm using Minio as a locally hosted (but outside my kubernetes cluster) S3 endpoint. I validated I can hit Minio via S3 compatible client in the same cluster and namespace

### To Reproduce

Deploy fluent bit operator helm chart with ES and S3 enabled as outputs

### Expected behavior

Both ES and S3 should work as outputs from fluent-bit

### Your Environment

```markdown
- Fluent Operator version: 3.2.0
- Container Runtime: containerd / kubernetes
- Operating system: Linux / Ubuntu
- Kernel version: 6.8.0-48 generic
```

### How did you install fluent operator?

Helm Chart

### Additional context

This is my values.yaml configuration

```
Kubernetes: true
containerRuntime: containerd
fluentbit:
additionalVolumes: []
additionalVolumesMounts: []
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/edge
operator: DoesNotExist
annotations: {}
args: []
command: []
crdsEnable: true
disableLogVolumes: false
enable: true
envVars: []
filter:
containerd:
enable: true
kubernetes:
annotations: false
enable: true
labels: false
multiline:
emitterMemBufLimit: 120
enable: false
keyContent: log
parsers:
- go
- python
- java
systemd:
enable: true
hostNetwork: false
image:
repository: ghcr.io/fluent/fluent-operator/fluent-bit
tag: v3.1.8
imagePullSecrets: []
initContainers: []
input:
fluentBitMetrics: {}
nodeExporterMetrics: {}
systemd:
enable: true
includeKubelet: true
path: /var/log/journal
pauseOnChunksOverlimit: 'off'
storageType: filesystem
stripUnderscores: 'off'
systemdFilter:
enable: true
filters: []
LimitNOFILE: '20000'
bufferMaxSize: 4GB
tail:
bufferMaxSize: 4GB
enable: true
memBufLimit: 100MB
path: /var/log/containers/*.log
pauseOnChunksOverlimit: 'off'
readFromHead: false
refreshIntervalSeconds: 30
skipLongLines: true
storageType: filesystem
kubeedge:
enable: false
prometheusRemoteWrite:
host:
port:
labels: {}
logLevel: ''
namespaceFluentBitCfgSelector: {}
nodeSelector: {}
output:
es:
bufferSize: 4GB
enable: true
host: elasticsearch.logging.svc.cluster.local
logstashPrefix: ks-logstash-log
port: 9200
traceError: true
logstashFormat: true
retry: true
retryBackoff: 10s
retryLimit: false
retryWait: 5
storageType: filesystem
suppressTypeName: 'On'
kafka:
brokers:
enable: false
topics: ks-log
loki:
enable: false
host: 127.0.0.1
httpPassword: mypass
httpUser: myuser
port: 3100
tenantID: ''
opensearch: {}
opentelemetry: {}
prometheusMetricsExporter: {}
stackdriver: {}
stdout:
enable: true
s3:
aws_key_id:
aws_secret_key:
bucket: logarchive
compression: gzip
enable: true
endpoint: http://192.168.1.85:9000
force_path_style: true
path: logs/${TAG}/%Y/%m/%d/
region: us-east-1
s3_key_format: ${TAG}/%Y/%m/%d/%H/%M/%S
tls_verify: false
total_file_size: 5M
upload_timeout: 60s
parsers:
javaMultiline:
enable: false
podSecurityContext: {}
priorityClassName: ''
rbacRules: {}
resources:
limits:
cpu: 500m
memory: 1024Mi
requests:
cpu: 10m
memory: 25Mi
schedulerName: ''
secrets: []
securityContext: {}
service:
storage:
checksum: 'off'
deleteIrrecoverableChunks: 'on'
maxChunksUp: 128
metrics: 'on'
path: /tmp/fluent/logging/
sync: normal
serviceAccountAnnotations: {}
serviceMonitor:
enable: false
interval: 30s
metricRelabelings: []
path: /api/v2/metrics/prometheus
relabelings: []
scrapeTimeout: 10s
secure: false
tlsConfig: {}
backlogMemLimit: 256MB
tolerations:
- operator: Exists
fluentd:
crdsEnable: true
enable: false
envVars: []
extras: {}
forward:
port: 24224
image:
repository: ghcr.io/fluent/fluent-operator/fluentd
tag: v1.17.0
imagePullSecrets: []
logLevel: ''
mode: collector
name: fluentd
output:
es:
buffer:
enable: true
path: /tmp/fluent/logging/es
type: file
maxBytes: 4GB
enable: false
host: elasticsearch-logging-data.kubesphere-logging-system.svc
logstashPrefix: ks-logstash-log
port: 9200
kafka:
brokers: >-
my-cluster-kafka-bootstrap.default.svc:9091,my-cluster-kafka-bootstrap.default.svc:9092,my-cluster-kafka-bootstrap.default.svc:9093
buffer:
enable: true
path: /tmp/fluent/logging/kafka
type: file
maxBytes: 4GB
enable: false
topicKey: kubernetes_ns
opensearch: {}
podSecurityContext: {}
port: 24224
priorityClassName: ''
replicas: 1
resources:
limits:
cpu: 500m
memory: 500Mi
requests:
cpu: 100m
memory: 128Mi
schedulerName: ''
securityContext: {}
watchedNamespaces:
- kube-system
- default
fullnameOverride: ''
nameOverride: ''
namespaceOverride: ''
operator:
annotations: {}
container:
repository: kubesphere/fluent-operator
tag: v3.2.0
disableComponentControllers: fluentd
enable: true
extraArgs: []
imagePullSecrets: []
initcontainer:
repository: docker
resources:
limits:
cpu: 500m
memory: 1024Mi
requests:
cpu: 250m
memory: 512Mi
tag: '20.10'
labels: {}
logPath:
containerd: /var/log
nodeSelector: {}
podSecurityContext: {}
priorityClassName: ''
resources:
limits:
cpu: 500m
memory: 1024Mi
requests:
cpu: 100m
memory: 512Mi
securityContext: {}
tolerations: []
```

Here are the logs I see on fluent-bit pod startup, indicating S3 isn't being parsed as an output, it seems

```
Container: fluent-bit
output
Connected
[2024/11/05 17:28:10] [ info] [output:es:es.0] worker #0 started
[2024/11/05 17:28:10] [ info] [output:es:es.0] worker #1 started
[2024/11/05 17:28:10] [ info] [output:stdout:stdout.1] worker #0 started
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Helm chart deployment using the supplied values.yaml and inspect the generated Fluent Bit configuration and pod startup logs. Done means the generated configuration includes the S3 output and Fluent Bit starts both Elasticsearch and S3 output workers successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.