fluent / fluent/fluent-bit

kubernetes filter: SIGSEGV when two filter instances enable pod association (Use_Pod_Association) — regression in 5.0.5, still present in 5.0.9

Open
#12,152 0 comments 0 reactions 0 assignees View on GitHub
status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## Bug Report

**Describe the bug**

Fluent Bit crashes with SIGSEGV when **two `kubernetes` filter instances** (matching different tags) both enable pod association (`Use_Pod_Association On`, the alias of `aws_use_pod_association`). The crash occurs in the TLS session-creation path of the pod-association / entity connection to the CloudWatch agent endpoint (`:4311`).

This is a **regression introduced in v5.0.5**: the identical configuration runs indefinitely on v5.0.3, and crashes on v5.0.5 through v5.0.9 (latest at the time of writing).

Two distinct triggers produce the same crash:

1. **Endpoint loss**: when the pod-association endpoint becomes unreachable (CloudWatch agent pod restart / node replacement), Fluent Bit crashes within minutes.
2. **Steady state**: even with the endpoint healthy and the node stable for 15+ hours, Fluent Bit crashes roughly every 25 minutes (min ~2 min, max ~1.5 h). Every crash lands on the 60-second connection-refresh cycle aligned to container start time (container started at hh:26:31 → all errors/crashes at hh:mm:32), suggesting corrupted shared state is hit by the periodic reconnect rather than the reconnect itself being the root cause.

**To Reproduce**

- Rubular link if applicable: N/A
- Example log message if applicable:

```
[2026/07/23 07:38:32.200] [error] [openssl] could create new SSL context
[2026/07/23 07:38:32.200] [error] [tls] could not create TLS session for tcp://172.20.202.50:4311
[2026/07/23 07:38:32] [engine] caught signal (SIGSEGV)
```

(The first message is the known mislabeled/typo'd one emitted from `tls_session_create()` when `SSL_new()` fails — see #4666. `SSL_new()` failing is normally very rare, which itself points to a corrupted/freed `SSL_CTX`. The container exits with code 133 = 128+SIGTRAP because the crash handler traps after logging, on ARM64.)

- Steps to reproduce the problem:

We could not reduce this to a standalone repro outside Kubernetes (the pod-association path requires the CloudWatch agent entity endpoint), but the in-cluster reproduction is deterministic and was confirmed across dozens of crashes on two independent EKS clusters:

1. Deploy the `amazon-cloudwatch-observability` EKS addon (v6.2.0 or v6.3.0, i.e. Fluent Bit 5.0.5) with a customized Fluent Bit configuration containing **two** `kubernetes` filter instances, both with pod association enabled:

```ini
# file 1 (application logs)
[FILTER]
Name kubernetes
Match application.*
Use_Kubelet On
Buffer_Size 0
Use_Pod_Association On

# file 2 (dataplane logs) — the second instance that triggers the crash
[FILTER]
Name kubernetes
Match dataplane.tail.*
Use_Kubelet On
Buffer_Size 0
Use_Pod_Association On
```

2. Delete the CloudWatch agent pods (`kubectl delete pod -n amazon-cloudwatch -l app.kubernetes.io/component=amazon-cloudwatch-agent`) so the pod-association endpoint disappears briefly.
3. Fluent Bit crashes within minutes (in steady state it also crashes on its own, roughly every 25 minutes).

Single-variable isolation results (same cluster, same test):

| Configuration | Fluent Bit | Result |
|---|---|---|
| 1 filter instance with pod association (default addon config) | 5.0.5 | No crash (endpoint loss ×3 + steady state — errors handled cleanly) |
| 2 filter instances, both `Use_Pod_Association On` | 5.0.3 | No crash |
| 2 filter instances, both `Use_Pod_Association On` | 5.0.5 | Crash (6.4 min after endpoint loss; ~25 min cycle in steady state) |
| 2 filter instances, both `Use_Pod_Association On` | 5.0.9 | Crash (same signature, within minutes) |
| Same, but `Use_Pod_Association` removed from **one** of the two filters | 5.0.5 | No crash under the identical endpoint-loss test |

The last two rows differ by exactly one line of configuration.

**Expected behavior**

Connection failures to the pod-association endpoint should be handled as recoverable errors (as they are in v5.0.3, and in single-filter-instance configurations on v5.0.5+), not crash the process.

**Screenshots**

N/A

**Your Environment**

* Version used: Fluent Bit 5.0.5 and 5.0.9 (crash), 5.0.3 (no crash) — via `aws-for-fluent-bit` 3.4.0 / 3.4.3 / 3.4.9 and 3.3.0
* Configuration: see excerpt above; full config is the `amazon-cloudwatch-observability` addon default plus a second `kubernetes` filter with `Use_Kubelet On` + `Use_Pod_Association On` on a separate tag
* Environment name and version (e.g. Kubernetes? What version?): Amazon EKS 1.34 / 1.35, deployed via the `amazon-cloudwatch-observability` EKS addon (v5.4.0 / v6.2.0 / v6.3.0)
* Server type and version: AWS Graviton (ARM64), Spot and On-Demand
* Operating System and version: Bottlerocket OS
* Filters and plugins: `filter_kubernetes` (×2, with `Use_Kubelet` + `Use_Pod_Association`), `filter_aws` (`Enable_Entity true`), `filter_log_to_metrics`, `out_cloudwatch_logs` (`add_entity true`), `out_prometheus_exporter` (log_to_metrics / prometheus_exporter were ruled out by isolation testing)

**Additional context**

Suspected root cause: v5.0.5 added TLS session/context destroy calls to `filter_kubernetes` as part of the memory-leak fixes (#11730, #11731, #11738). With two filter instances running pod association concurrently, one instance appears to destroy TLS/pod-association state still referenced by the other (use-after-free), which later manifests as `SSL_new()` failure followed by SIGSEGV on the periodic reconnect path. This would also explain why v5.0.3 — which leaked instead of destroying — never crashes.

Verified workaround for anyone hitting this signature: ensure **only one** `kubernetes` filter instance enables pod association. Removing `Use_Pod_Association On` from the second filter eliminates the crash while keeping metadata enrichment intact.

We are unblocked by the workaround, so this report is primarily to document the issue for other users searching for this crash signature, and to flag the suspected use-after-free (which may be reachable through paths other than our trigger). Happy to test patched builds, or answer questions.

Contributor guide

Open the contributing guide

Research direction

Start with the filter_kubernetes pod-association TLS path and tls_session_create(), then compare the v5.0.3 behavior with the TLS/session-destroy changes associated with #11730, #11731, and #11738. Reproduce using two filters with Use_Pod_Association On and endpoint loss; done means connection failures recover without SIGSEGV or corrupted shared state.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, c, kubernetes
Domain
backend, cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.