fluent / fluent/fluent-bit

[Regression] 4.2.4: opensearch output sustained flush failures, ~60% ingestion rate drop vs 4.2.3

Open
#11,799 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

## Bug Report

**Describe the bug**

After upgrading from `cr.fluentbit.io/fluent/fluent-bit:4.2.3` to `4.2.4`, the `opensearch` output plugin produces sustained `failed to flush chunk` warnings and `cannot be retried` errors against a healthy OpenSearch 2.19.4 backend. Ingestion rate drops from ~480 docs/min (baseline at 4.2.3) to ~200 docs/min, with `Retry_Limit 3` exhausted and chunks discarded. OpenSearch itself logs no errors and accepts writes from other clients without issue. Rolling back the DaemonSet image to `4.2.3` (no other config change) immediately restores normal behavior: 0 flush errors, ingestion rate recovers to ~561 docs/min.

**To Reproduce**

Steps:
1. Run Fluent Bit `4.2.3` as a DaemonSet on a Kubernetes cluster, tailing `/var/log/containers/*.log` and shipping to an in-cluster OpenSearch via the `opensearch` output. Confirm steady ingestion.
2. Change only the image tag in the DaemonSet to `4.2.4` and apply.
3. After all DaemonSet pods roll to the new image, observe Fluent Bit logs filling with:

```
[engine] failed to flush chunk '1-..flb', retry in N seconds: task_id=X, input=tail.0 > output=opensearch.0 (out_id=0)
[engine] chunk '1-..flb' cannot be retried: task_id=X, input=tail.0 > output=opensearch.0
```

at a rate of 16+ warnings per 30 seconds (sustained, not a transient burst). Querying the OpenSearch daily index shows the doc-arrival rate roughly halved versus the same time-of-day at 4.2.3.

4. Roll the DaemonSet image back to `4.2.3` (no other change). Errors stop within one rollout, ingestion rate recovers immediately.

**Expected behavior**

`opensearch` output should ship chunks as efficiently in `4.2.4` as in `4.2.3`. No persistent flush failures against a healthy backend.

**Your Environment**

* **Version used**: Fluent Bit `4.2.4` (regressed from `4.2.3`)
* **Image**: `cr.fluentbit.io/fluent/fluent-bit:4.2.4` (official, amd64)
* **Configuration (output section)**:
```ini
[OUTPUT]
Name opensearch
Match kube.*
Host opensearch.logging.svc.cluster.local
Port 9200
Index fluent-bit-kube
Type _doc
Suppress_Type_Name On
Logstash_Format On
Logstash_Prefix fluent-bit-kube
Time_Key @timestamp
Retry_Limit 3
tls Off
```
* **Inputs/filters**: `tail` reading `/var/log/containers/*.log` + `kubernetes` filter (standard).
* **Kubernetes**: microk8s 1.35.0 (containerd://2.1.3), 8 nodes (Ubuntu 24.04 LTS).
* **OpenSearch**: `opensearchproject/opensearch:2.19.4`, single-node, plain HTTP on port 9200 (security plugin disabled), healthy throughout the regression. No relevant log entries on the OpenSearch side during the window. The cluster continues to accept writes from non-Fluent-Bit clients without issue.

**Additional context**

This is a real, measurable regression on a production homelab cluster, not a transient issue. Reproduced cleanly across all 8 Fluent Bit DaemonSet pods on different nodes simultaneously. Rollback to 4.2.3 is the only mitigation we've found.

Suspected scope based on the `v4.2.3...v4.2.4` diff (without bisecting):

- **PR #11519** (engine: use effective post-processor counts for output metrics) — modifies `flb_engine.c handle_output_event()` directly, the function processing every flush result, plus adds new task/route fields. The PR description states it's metrics-only, but the modifications are to the hot flush-handling path; a defect there could explain the observed retry/drop pattern.
- **PR #11693** (lib: monkey: upgrade to v1.8.8) — large library bump (286 additions) that may affect HTTP client behavior.

(Note: `4.2.3.1` is just a Dockerfile-only release with no runtime code change, so it's not a useful intermediate. As of writing, there is no `4.2.5` and the `4.2.x` branch appears effectively dormant — recent activity is on the `5.x` line, which we cannot adopt due to the separate `yyjson` parser incompatibility with non-JSON inputs.)

Happy to provide more telemetry, run a sandbox bisect if there's interest, or test a proposed fix.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.