fluent / fluent/fluent-bit

[ warn] [engine] failed to flush chunk

Open
#12,302 0 comments 0 reactions 0 assignees View on GitHub
status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

Hi,

Fluentbit 5.0.3 Windows version, installed as windows service.

Tail and Opensearch plugin is used. Operation is upsert.

Usual config was->log_level=warning, no trace_output

It was enough to see the errors -in other cases-
However there is nothing about why this "warning/error" occurs.
So even i changed it to below settings. It does not give any clue. Literally nothing.

Records are tried indefinitely. Long records are skipped.

Logs are written to Opensearch, it seems fine.
But there is a storage config.
My concern is, it might overflow the storage if such records can not be processed.

In which scenario, fluentbit refuses to log the issue with such config?

Note:
According the source codes, this happens because of memory or file handle issues.
Fluent-bit running under 200MB, plenty of GBs are available on the machine.
Handle count fluntuates around 1450. (Top limit is not 512)
I am not sure this hits either of those issues.
Maybe for the future release, in trace mode it should write the actual issue where it happens.

-Please ignore the format-
Service:
log_level=trace

Output:
Buffer_Size=False
Trace_Error=On
Trace_Output=On

[2026/08/17 12:57:16.059] [debug] [upstream] KA connection #968 to xxx:9200 is now available
[2026/08/17 12:57:16.059] [debug] [out flush] cb_destroy coro_id=0
[2026/08/17 12:57:16.059] [debug] [task] destroy task=00000185687C59C0 (task_id=1)
[2026/08/17 12:57:16.061] [debug] [out flush] cb_destroy coro_id=0
[2026/08/17 12:57:16.061] [debug] [task] destroy task=00000185687A9540 (task_id=0)
[2026/08/17 12:57:17.029] [debug] [task] created task=00000185687A9540 id=0 OK

[2026/08/17 12:57:17.126] [debug] [output:opensearch:opensearch.3] HTTP Status=200 URI=/_bulk/_bulk
[2026/08/17 12:57:17.128] [debug] [upstream] KA connection #5700 to xxx:9200 is now available
[2026/08/17 12:57:17.128] [debug] [out flush] cb_destroy coro_id=0
[2026/08/17 12:57:17.128] [debug] [retry] new retry created for task_id=0 attempts=1
[2026/08/17 12:57:17.130] [ warn] [engine] failed to flush chunk '5772-1785401914.324505100.flb', retry in 6 seconds: task_id=0, input=storage_backlog.48 > output=opensearch.3 (out_id=3)

[2026/08/17 13:02:27.301] [debug] [upstream] KA connection #1224 to xxx:9200 has been assigned (recycled)
[2026/08/17 13:02:27.311] [debug] [http_client] not using http_proxy for header
[2026/08/17 13:02:27.392] [debug] [output:opensearch:opensearch.3] HTTP Status=200 URI=/_bulk/_bulk
[2026/08/17 13:02:27.394] [debug] [upstream] KA connection #1224 to xxx:9200 is now available
[2026/08/17 13:02:27.394] [debug] [out flush] cb_destroy coro_id=13
[2026/08/17 13:02:27.394] [debug] [retry] re-using retry for task_id=0 attempts=14
[2026/08/17 13:02:27.395] [ warn] [engine] failed to flush chunk '5772-1785401914.324505100.flb', retry in 12 seconds: task_id=0, input=storage_backlog.48 > output=opensearch.3 (out_id=3)

Contributor guide

Open the contributing guide

Research direction

Start with the [engine] failed to flush chunk warning and compare it with the preceding OpenSearch HTTP 200 and retry messages in the report. Trace the flush failure path for storage_backlog.48 to determine why the underlying memory or file-handle cause is absent, and consider the issue done when trace mode reports the actual failure reason.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.