fluent-bit k8s events input causes crashes (SIGSEGV) in the in_kubernetes_events while processing the Kubernetes watch stream
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
## Bug Report
**Describe the bug**
in_kubernetes_events crashes with a SIGSEGV every few hours while reading the Kubernetes watch stream.
The crash is in process_http_chunk(), which calls strpbrk(c->resp.payload, "\r\n") without checking resp.payload first. By the time it runs, that pointer can be pointing at memory that has already been freed.
Why it happens: after a watch chunk has been fully read, resp.payload_size is set to 0 but resp.payload is left pointing at the old buffer. The next read grows the buffer with flb_http_buffer_increase(), and the realloc inside it can move the buffer and free the old one. The code that updates resp.payload to the new location only runs when payload_size is greater than 0 (flb_http_client.c:1225 and :1245), so when it is 0 the pointer never gets updated and is left pointing at the freed memory. process_http_chunk() then reads through that stale pointer and crashes.
**To Reproduce**
- Rubular link if applicable: N/A
- Example log message if applicable:
```
[engine] caught signal (SIGSEGV)
#0 process_http_chunk() at plugins/in_kubernetes_events/kubernetes_events.c:752
#1 k8s_events_collect() at plugins/in_kubernetes_events/kubernetes_events.c:920
#2 co_init() at lib/monkey/deps/flb_libco/amd64.c:117
```
- Steps to reproduce the problem:
1. Run in_kubernetes_events against a Kubernetes API server and leave it watching events for a long time.
2. The crash happens after a watch chunk is fully read (so payload_size becomes 0) and the next read grows the buffer and moves it in memory, which leaves resp.payload pointing at freed memory. It depends on timing, so in practice it shows up about once every few hours per pod.
3. You can also hit it on demand: call process_http_chunk() with an empty response (resp.payload == NULL, payload_size == 0), which is the state a stream starts in. Running that under AddressSanitizer reports a heap-use-after-free read in process_http_chunk, pointing back at the realloc in flb_http_buffer_increase.
**Expected behavior**
in_kubernetes_events processes the watch stream without crashing; an empty or already-consumed response is a no-op, not a segfault.
**Screenshots**
N/A
**Your Environment**
Version used: 5.0.9 and 5.1.0 (also present on current master)
Configuration:
[INPUT]
Name kubernetes_events
Tag k8s_events
DB /var/log/flb_kube_k8s_events.db
interval_sec 30
storage.type filesystem
Kube_Request_Limit 200
**Additional context**
We've seen this happen on and off across several production Kubernetes clusters. Each time, the fluent-bit-k8s-events pod dies (exit 134) and stops shipping Kubernetes events until it restarts. It looks like the same problem reported in #9543 ("Kubernetes Events Input Segfault"), which was closed without a fix.
Contributor guide
Research direction
Start with process_http_chunk() in plugins/in_kubernetes_events/kubernetes_events.c and inspect the response handling around flb_http_buffer_increase() in lib/flb_http_client.c, especially the cited lines 1225 and 1245. Reproduce the empty-response case under AddressSanitizer and exercise a Kubernetes watch stream; done means the stream survives buffer growth and empty or consumed responses no longer crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, kubernetes
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100