[critical] fluent-bit killed by OOM when reaching cpu limit
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
## Bug Report
**Describe the bug**
fluent-bit when receiving quite high load, after reaching its CPU limit has a memory usage increase and then is killed by k8s with 137 OOMKilled.
**To Reproduce**
- Run fluent-bit on k8s with opentelemetry input
- Run some load tests (i.e. using Grafana k6) sending otel logs via GRPC
- Observe cpu and mem stats of the containers (i.e. watch -n 1 kubectl -n namespace top pods)
**Expected behavior**
Load tests pass at most with some dropped iterations due to lack of performance. fluent-bit lives for entire test time.
**Your Environment**
* Version used: 5.0.3
* Configuration:
```
service:
flush: 1
grace: 10
log_level: info
hot_reload: on
http_server: on
health_check: on
http_port: 2020
storage.path: /fluent-bit/storage
storage.metrics: on
storage.sync: full
storage.max_chunks_up: 16
storage.backlog.mem_limit: 10MB
storage.backlog.flush_on_shutdown: on
storage.delete_irrecoverable_chunks: on
scheduler.base: 5
scheduler.cap: 60
pipeline:
inputs:
- name: opentelemetry
port: 4318
storage.type: filesystem
storage.pause_on_chunks_overlimit: off
http_server.buffer_max_size: 10MB
http_server.buffer_chunk_size: 512KB
filters:
- name: stdout
match: 'v1_logs'
- name: modify
match: 'v1_logs'
Condition:
- Key_Does_Not_Exist type
Set: type type_unknown
- name: rewrite_tag
match: 'v1_logs'
rule: $type ^(.+)$ $1 true
emitter_name: type_emitter
- name: rewrite_tag
match: 'type_unknown'
emitter_storage.type: filesystem
rule: $type ^(.+)$ $1 true
- name: log_to_metrics
match: 'v1_logs'
metric_mode: counter
metric_name: logs_total
metric_description: "Metric description"
tag: total_metrics
label_field: source
outputs:
- name: s3
match_regex: 'v1_logs'
retry_limit: 5
workers: 1
bucket: "bucket_name_1"
region: "region"
endpoint: "endpoint"
upload_timeout: 1m
s3_key_format: /$TAG/%Y-%m-%d-%H-%M-%S-$UUID.log
total_file_size: 1MB
json_date_key: false
use_put_object: true
- name: s3
match_regex: 'v1_logs'
retry_limit: 5
workers: 1
bucket: "bucket_name_2"
region: "region"
endpoint: "endpoint"
upload_timeout: 1m
s3_key_format: /some_prefix/%Y-%m-%d-%H-%M-%S-$UUID.log
total_file_size: 1MB
json_date_key: false
use_put_object: true
- name: prometheus_exporter
match_regex: '^(v1_logs|total_metrics)$'
port: 2021
```
* Environment name and version (e.g. Kubernetes? What version?): k8s v1.31.9
* Server type and version:
* Operating System and version: workers running Ubuntu 22.04 LTS
* fluent-bit is run as a statefulset, 2 pods, 10Gi volumes, 1 cpu limit, 1 Gi memory limit for each replica.
* Load (simultaneous scenarios, 15mins):
** 300kB (10 otel logs), 6 rps
** 2kB (1 otel log), 7 rps
** 680kB (345 otel logs), 32 rps
**Additional context**
* during many test runs I've tried to tune many memory parameters, max active chunks count, however with no difference.
* same configuration run with fluent-bit 4.2.2 or 4.2.4 does not get killed and withstands the load.
Contributor guide
Assessment
This issue has not been assessed yet.