fluent / fluent/fluent-bit

s3 output Buffer Full Even with buffer path is empty too

Open
#12,149 3 comments 0 reactions 0 assignees View on GitHub
status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## Bug Report

S3 Counter Underflow Error

Ran with fluent-bit 5.0.7 and 5.0.9 both returning the error in sometime though buffer path not filled

[2026/07/23 17:35:18.363] [ warn] [output:s3:s3.0] Could not buffer chunk. Data order preservation will be compromised
[2026/07/23 17:35:18.363] [error] [output:s3:s3.0] Buffer is full: current_buffer_size=18446744073684963328, new_data=9272, store_dir_limit_size=2000000000 bytes

We have added below parameters to get the retried chunks get deleted post retry exceeded we are getting buffer full error though file path is empty

retry_limit 5
retry_exhausted_action delete
* Version used: 5.0.7 ,5.0.9

[SERVICE]
Flush 120
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server ${HTTP_SERVER}
HTTP_Listen 0.0.0.0
HTTP_Port ${HTTP_PORT}
storage.path /var/fluent-bit/state/flb-storage/
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 50M
storage.max_chunks_up 512
[INPUT]
Name fluentbit_metrics
Tag fluentbit_metrics
scrape_interval 30
[INPUT]
Name tail
Tag kubernetes.logs.*
Path /var/log/containers/*.log
Docker_Mode On
Docker_Mode_Flush 5
Docker_Mode_Parser container_firstline
Parser docker
DB /var/fluent-bit/state/flb_container.db
Skip_Long_Lines On
Refresh_Interval 5
Rotate_Wait 30
storage.type filesystem
Read_from_Head false
[FILTER]
Name kubernetes
Match kubernetes.logs.*
tls.verify Off
Kube_URL [https://kubernetes.default.svc.cluster.local:443](https://kubernetes.default.svc.cluster.local/)
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kubernetes.logs.var.log.containers
Merge_Log On
Labels Off
Annotations Off
Buffer_Size 0
Use_Kubelet true
Kubelet_Port 10250
[FILTER]
Name nest
Match kubernetes.logs.*
Operation lift
Nested_under kubernetes
[FILTER]
Name modify
Match kubernetes.logs.*
Add tag docker.all
[OUTPUT]
Name s3
Match_regex ^(kubernetes.logs.*)$
Bucket ${S3_BUCKET}
Region us-east-1
endpoint ${S3_ENDPOINT}
compression gzip
s3_key_format /logs/v1_%Y%m%d%H%M%S_$UUID.gz
Upload_chunk_size 50M
static_file_path On
use_put_object Off
store_dir /var/log/fluentbit/log/s3/
store_dir_limit_size 2GB
retry_limit 5
retry_exhausted_action delete
[OUTPUT]
Name prometheus_exporter
match fluentbit_metrics
host ::
port 2022

BufferPath:

/var/log/fluentbit/log/s3$ ls -lhtr
var/log/fluentbit/log/s3$ ls -lhtr
total 16K
drwxr-xr-x 2 root root 4.0K Jul 17 17:07 quarantine
drwxr-xr-x 2 root root 4.0K Jul 17 17:07 multipart_upload_metadata
drwxr-xr-x 2 root root 4.0K Jul 20 15:09 2026-07-20T15:09:42
drwxr-xr-x 2 root root 4.0K Jul 20 15:09 2026-07-17T17:07:11

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the S3 output failure with the configuration and log sequence in the issue, focusing on the buffer-size accounting despite an empty store directory. Trace the S3 output buffering and retry-exhaustion path; done means the reported counter-underflow and false "Buffer is full" condition no longer occur under this configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, c
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.