in_prometheus_scrape never reuses TCP connections; root cause also affects flb_input_upstream_set()
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
## Bug Report
_Investigation and this writeup were AI-assisted (Claude Code)._
**Describe the bug**
`prometheus_scrape` opens a brand new TCP connection and tears it down
(FIN) on every single scrape cycle, regardless of `net.keepalive`.
Confirmed via packet capture: a full SYN/SYN-ACK/ACK handshake followed
by FIN immediately after each response, every `scrape_interval`, never
a reused connection.
This is actually two separate, independently-necessary bugs:
- **Bug 1: `in_prometheus_scrape` never calls `flb_input_upstream_set()`**
`plugins/in_prometheus_scrape/prom_scrape.c` calls `flb_upstream_create()`
but never calls `flb_input_upstream_set()`, so `net.*` properties are
parsed into `ins->net_setup` but never applied to the plugin's upstream.
Same class of bug already fixed for `in_kubernetes_events` (#11188) and
`in_calyptia_fleet` (#10998).
- **Bug 2: `flb_input_upstream_set()` itself is missing the `FLB_IO_TCP_KA` flag**
Adding the call above is not sufficient on its own (verified — see below).
`flb_upstream_conn_get()` sets a new connection's initial `recycle` value
from `flb_stream_is_keepalive()`, which checks the `FLB_IO_TCP_KA` bit on
`stream->flags` — not `net.keepalive` directly.
`flb_output_upstream_set()` (`src/flb_output.c`) sets this flag correctly:
```c
if (ins->net_setup.keepalive == FLB_TRUE) {
flags |= FLB_IO_TCP_KA;
}
flb_stream_enable_flags(&u->base, flags);
```
`flb_input_upstream_set()` (`src/flb_input.c`) only does the `net.*`
memcpy — it never sets `FLB_IO_TCP_KA`. So `conn->recycle` starts
`FALSE` on every new connection, and `flb_upstream_conn_release()`'s
`conn->recycle == FLB_TRUE` check always fails, forcing a fresh
connection every time regardless of `net.keepalive`.
This affects every input plugin that calls `flb_input_upstream_set()` —
likely including `in_kubernetes_events` and `in_calyptia_fleet` despite
their existing fixes, though I have not independently reproduced that;
it's an inference from shared code, not a verified repro.
**To Reproduce**
- Steps to reproduce the problem:
1. Configure `prometheus_scrape` against any HTTP/1.1 keep-alive-capable
endpoint.
2. Capture loopback traffic:
```
tcpdump -i lo -n -tttt 'tcp port '
```
3. Observe a new SYN/FIN pair every `scrape_interval`, never a reused
connection.
**Expected behavior**
Connection is established once and reused across scrape cycles, per
`net.keepalive` (default: on).
**Screenshots**
N/A — see packet-capture description above.
**Your Environment**
* Version used: v5.0.7
* Configuration: `prometheus_scrape` input scraping two independent
HTTP/1.1 endpoints on loopback
* Environment name and version: N/A (not Kubernetes)
* Server type and version: Amazon Linux 2023, x86_64
* Operating System and version: Amazon Linux 2023, x86_64
* Filters and plugins: `in_prometheus_scrape`
**Additional context**
Fix verified locally, tested in 4 configurations (v5.0.7, x86_64):
| Configuration | Result |
|---------------------------|----------------------------------|
| Unpatched (baseline) | New connection every cycle |
| Bug 1 fix only | New connection every cycle |
| Bug 2 fix only | New connection every cycle |
| Both fixes together | Connection created once, reused |
Confirmed via `-vv` debug log: with both fixes, connections show
`"KA connection #N ... is connected"` once, then
`"has been assigned (recycled)"` on every subsequent scrape. With either
fix alone, no `"KA connection"` line ever appears for these endpoints.
I'll open two PRs against this issue: one for Bug 1 (scoped to
`in_prometheus_scrape`, mirrors the existing `in_kubernetes_events` fix)
and one for Bug 2 (core `flb_input_upstream_set()` fix, broader impact —
worth extra scrutiny and possibly a look at the other two plugins).
Contributor guide
Research direction
Start with plugins/in_prometheus_scrape/prom_scrape.c and src/flb_input.c, comparing flb_input_upstream_set() with flb_output_upstream_set(); trace flb_upstream_conn_get() and flb_upstream_conn_release(). Reproduce with the supplied prometheus_scrape configuration and tcpdump command. Done means both reported paths work together so net.keepalive reuses connections across scrape cycles, confirmed by packet capture or -vv logs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, prometheus
- Domain
- backend, networking, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100