fluent / fluent/fluent-bit

out_prometheus_exporter ignores HTTP server worker and buffer options

Open Beginner friendly
#12,110 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

## Bug Report

**Describe the bug**

`out_prometheus_exporter` parses the generic HTTP server configuration, but
`prom_http_server_create()` does not pass `buffer_chunk_size` or `workers` to
`flb_http_server_init_with_options()`.

Consequently, `http_server.workers` is accepted without creating the requested worker
runtime. Under a large metrics workload, synchronous metrics decoding and Prometheus
text encoding in `cb_prom_flush()` can occupy the caller event loop and delay HTTP
response handling.

The omission is present in Fluent Bit 5.0.2, current `5.0`, and current `master`.

**To Reproduce**

1. Configure the exporter with explicit HTTP options:

```yaml
service:
flush: 1

pipeline:
inputs:
- name: fluentbit_metrics
scrape_interval: 1

outputs:
- name: prometheus_exporter
match: "*"
host: 127.0.0.1
port: 2021
http_server.buffer_chunk_size: 64K
http_server.workers: 2
```

2. Start Fluent Bit and inspect its threads.
3. Observe that no HTTP worker runtime or listener worker threads are created.
4. Apply the two missing assignments and repeat.
5. Observe two HTTP worker threads and a non-null worker runtime.

A deterministic runtime test reproduces the defect without timing thresholds:

```c
server.buffer_chunk_size == ins->http_server_config->buffer_chunk_size
server.workers == 2
server.runtime != NULL
```

All three checks fail as applicable before the fix and pass afterward.

**Expected behavior**

The parsed `http_server.buffer_chunk_size` and `http_server.workers` settings should
reach the embedded HTTP server.

**Your Environment**

- Version used: 5.0.2 and master at `fd5ea1f0d503`
- Operating systems: embedded Linux arm64 and Debian 12 x86_64 test container
- Plugins: `process_exporter_metrics`, `fluentbit_metrics`,
`prometheus_exporter`

**Additional context**

With approximately 1500 synthetic processes and a 2.47 MB metrics chunk, one
correlated reproduction showed:

```text
request TTFB: 673.957 ms
cb_prom_flush total: 746.633 ms
cb_prom_flush decode: 740.716 ms
HTTP handler total: 8.083 ms
HTTP payload mutex wait: ~0.002-0.003 ms
```

After applying the option-propagation fix and configuring two HTTP workers:

```text
requests: 762
worst TTFB: 46 ms
concurrent flush: ~954 ms
```

The fix does not reduce encoding CPU time. It makes the configured HTTP worker runtime
effective, allowing request handling to proceed independently.

A separate follow-up enhancement can default the exporter to one output worker so
`cb_prom_flush()` does not run on the main engine event loop even when HTTP workers are
not explicitly configured.

Contributor guide

Open the contributing guide

Research direction

Start in the out_prometheus_exporter implementation at prom_http_server_create() and trace the parsed http_server configuration into flb_http_server_init_with_options(). Run the supplied configuration or deterministic runtime checks; done means buffer_chunk_size and workers are propagated and server.runtime is non-null with workers=2.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, prometheus
Domain
backend, observability
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.