fluent / fluent/fluent-bit

in_prometheus_scrape never reuses TCP connections; root cause also affects flb_input_upstream_set()

Open
#12,094 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

## Bug Report

_Investigation and this writeup were AI-assisted (Claude Code)._

**Describe the bug**

`prometheus_scrape` opens a brand new TCP connection and tears it down
(FIN) on every single scrape cycle, regardless of `net.keepalive`.
Confirmed via packet capture: a full SYN/SYN-ACK/ACK handshake followed
by FIN immediately after each response, every `scrape_interval`, never
a reused connection.

This is actually two separate, independently-necessary bugs:

- **Bug 1: `in_prometheus_scrape` never calls `flb_input_upstream_set()`**
`plugins/in_prometheus_scrape/prom_scrape.c` calls `flb_upstream_create()`
but never calls `flb_input_upstream_set()`, so `net.*` properties are
parsed into `ins->net_setup` but never applied to the plugin's upstream.
Same class of bug already fixed for `in_kubernetes_events` (#11188) and
`in_calyptia_fleet` (#10998).

- **Bug 2: `flb_input_upstream_set()` itself is missing the `FLB_IO_TCP_KA` flag**
Adding the call above is not sufficient on its own (verified — see below).
`flb_upstream_conn_get()` sets a new connection's initial `recycle` value
from `flb_stream_is_keepalive()`, which checks the `FLB_IO_TCP_KA` bit on
`stream->flags` — not `net.keepalive` directly.

`flb_output_upstream_set()` (`src/flb_output.c`) sets this flag correctly:

```c
if (ins->net_setup.keepalive == FLB_TRUE) {
flags |= FLB_IO_TCP_KA;
}
flb_stream_enable_flags(&u->base, flags);
```

`flb_input_upstream_set()` (`src/flb_input.c`) only does the `net.*`
memcpy — it never sets `FLB_IO_TCP_KA`. So `conn->recycle` starts
`FALSE` on every new connection, and `flb_upstream_conn_release()`'s
`conn->recycle == FLB_TRUE` check always fails, forcing a fresh
connection every time regardless of `net.keepalive`.

This affects every input plugin that calls `flb_input_upstream_set()` —
likely including `in_kubernetes_events` and `in_calyptia_fleet` despite
their existing fixes, though I have not independently reproduced that;
it's an inference from shared code, not a verified repro.

**To Reproduce**

- Steps to reproduce the problem:
1. Configure `prometheus_scrape` against any HTTP/1.1 keep-alive-capable
endpoint.
2. Capture loopback traffic:
```
tcpdump -i lo -n -tttt 'tcp port '
```
3. Observe a new SYN/FIN pair every `scrape_interval`, never a reused
connection.

**Expected behavior**

Connection is established once and reused across scrape cycles, per
`net.keepalive` (default: on).

**Screenshots**

N/A — see packet-capture description above.

**Your Environment**

* Version used: v5.0.7
* Configuration: `prometheus_scrape` input scraping two independent
HTTP/1.1 endpoints on loopback
* Environment name and version: N/A (not Kubernetes)
* Server type and version: Amazon Linux 2023, x86_64
* Operating System and version: Amazon Linux 2023, x86_64
* Filters and plugins: `in_prometheus_scrape`

**Additional context**

Fix verified locally, tested in 4 configurations (v5.0.7, x86_64):

| Configuration | Result |
|---------------------------|----------------------------------|
| Unpatched (baseline) | New connection every cycle |
| Bug 1 fix only | New connection every cycle |
| Bug 2 fix only | New connection every cycle |
| Both fixes together | Connection created once, reused |

Confirmed via `-vv` debug log: with both fixes, connections show
`"KA connection #N ... is connected"` once, then
`"has been assigned (recycled)"` on every subsequent scrape. With either
fix alone, no `"KA connection"` line ever appears for these endpoints.

I'll open two PRs against this issue: one for Bug 1 (scoped to
`in_prometheus_scrape`, mirrors the existing `in_kubernetes_events` fix)
and one for Bug 2 (core `flb_input_upstream_set()` fix, broader impact —
worth extra scrutiny and possibly a look at the other two plugins).

Contributor guide

Open the contributing guide

Research direction

Start with plugins/in_prometheus_scrape/prom_scrape.c and src/flb_input.c, comparing flb_input_upstream_set() with flb_output_upstream_set(); trace flb_upstream_conn_get() and flb_upstream_conn_release(). Reproduce with the supplied prometheus_scrape configuration and tcpdump command. Done means both reported paths work together so net.keepalive reuses connections across scrape cycles, confirmed by packet capture or -vv logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, prometheus
Domain
backend, networking, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.