fluent / fluent/fluent-bit

[critical] fluent-bit killed by OOM when reaching cpu limit

Open
#11,745 0 comments 0 reactions 0 assignees View on GitHub
status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

## Bug Report

**Describe the bug**
fluent-bit when receiving quite high load, after reaching its CPU limit has a memory usage increase and then is killed by k8s with 137 OOMKilled.

**To Reproduce**
- Run fluent-bit on k8s with opentelemetry input
- Run some load tests (i.e. using Grafana k6) sending otel logs via GRPC
- Observe cpu and mem stats of the containers (i.e. watch -n 1 kubectl -n namespace top pods)

**Expected behavior**
Load tests pass at most with some dropped iterations due to lack of performance. fluent-bit lives for entire test time.

**Your Environment**
* Version used: 5.0.3
* Configuration:
```
service:
flush: 1
grace: 10
log_level: info
hot_reload: on
http_server: on
health_check: on
http_port: 2020
storage.path: /fluent-bit/storage
storage.metrics: on
storage.sync: full
storage.max_chunks_up: 16
storage.backlog.mem_limit: 10MB
storage.backlog.flush_on_shutdown: on
storage.delete_irrecoverable_chunks: on
scheduler.base: 5
scheduler.cap: 60

pipeline:
inputs:
- name: opentelemetry
port: 4318
storage.type: filesystem
storage.pause_on_chunks_overlimit: off
http_server.buffer_max_size: 10MB
http_server.buffer_chunk_size: 512KB
filters:
- name: stdout
match: 'v1_logs'

- name: modify
match: 'v1_logs'
Condition:
- Key_Does_Not_Exist type
Set: type type_unknown

- name: rewrite_tag
match: 'v1_logs'
rule: $type ^(.+)$ $1 true
emitter_name: type_emitter

- name: rewrite_tag
match: 'type_unknown'
emitter_storage.type: filesystem
rule: $type ^(.+)$ $1 true

- name: log_to_metrics
match: 'v1_logs'
metric_mode: counter
metric_name: logs_total
metric_description: "Metric description"
tag: total_metrics
label_field: source

outputs:
- name: s3
match_regex: 'v1_logs'
retry_limit: 5
workers: 1
bucket: "bucket_name_1"
region: "region"
endpoint: "endpoint"
upload_timeout: 1m
s3_key_format: /$TAG/%Y-%m-%d-%H-%M-%S-$UUID.log
total_file_size: 1MB
json_date_key: false
use_put_object: true

- name: s3
match_regex: 'v1_logs'
retry_limit: 5
workers: 1
bucket: "bucket_name_2"
region: "region"
endpoint: "endpoint"
upload_timeout: 1m
s3_key_format: /some_prefix/%Y-%m-%d-%H-%M-%S-$UUID.log
total_file_size: 1MB
json_date_key: false
use_put_object: true

- name: prometheus_exporter
match_regex: '^(v1_logs|total_metrics)$'
port: 2021
```
* Environment name and version (e.g. Kubernetes? What version?): k8s v1.31.9
* Server type and version:
* Operating System and version: workers running Ubuntu 22.04 LTS
* fluent-bit is run as a statefulset, 2 pods, 10Gi volumes, 1 cpu limit, 1 Gi memory limit for each replica.
* Load (simultaneous scenarios, 15mins):
** 300kB (10 otel logs), 6 rps
** 2kB (1 otel log), 7 rps
** 680kB (345 otel logs), 32 rps

**Additional context**
* during many test runs I've tried to tune many memory parameters, max active chunks count, however with no difference.
* same configuration run with fluent-bit 4.2.2 or 4.2.4 does not get killed and withstands the load.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.