fluent / fluent/fluentd

Reload race condition causes persistent 500 responses on `/aggregated_metrics` endpoint

Open
#5,310 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

### Describe the bug

When `fluentd` receives `SIGHUP` (e.g. via `systemctl reload fluentd.service`) to reload its
configuration, worker threads restart in-place. A new worker thread hits `Errno::EADDRINUSE`
when trying to bind its port because the old thread has not yet fully released the socket. The
crashing thread terminates silently, leaving the `fluent-plugin-prometheus` HTTP server on port
24231 running but unable to collect stats from the dead thread. All subsequent requests to the
`/aggregated_metrics` endpoint return HTTP 500 with body
`"Connection refused - connect(2) for 127.0.0.1:"`.

### To Reproduce

1. Install `fluent-package` with the following configuration (see **Your Configuration** below)
2. Start Fluentd: `systemctl start fluentd`
3. Send a reload signal: `systemctl reload fluentd.service` (sends SIGHUP)
4. Observe the warn log for `EADDRINUSE` on one of the worker threads
5. Request `http://localhost:24231/aggregated_metrics` — it returns HTTP 500 with body
`"Connection refused - connect(2) for 127.0.0.1:"`

### Expected behavior

After a reload (SIGHUP), all worker threads should successfully rebind their ports and Fluentd should continue serving `/aggregated_metrics` with HTTP 200. If a thread cannot rebind, it should either retry or fall back gracefully rather than crashing and leaving the supervisor in a broken state.

### Your Environment

```markdown
- Fluentd version: fluentd 1.19.1 (efdc4dca81c23480c9b55e13e55de6aa925b1cf5)
- Package version: fluent-package 6.0.1
- Operating system: Ubuntu 24.04.3
- Kernel version: 6.8.0-94-generic
```

### Your Configuration

```apache

workers 4

@type prometheus
bind 0.0.0.0
port 24231
metrics_path /metrics

@type prometheus_output_monitor

@type prometheus_monitor

@type http
bind 127.0.0.1
port 24224

@type json

@type null

```

### Your Error Log

```shell
2026-03-31 22:47:09 +0000 [warn]: #3 0.07s: Async::Task
| Task may have ended with unhandled exception.
| Errno::EADDRINUSE: Address already in use - bind
| → /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/wrapper.rb:152 in 'Socket#bind'
| /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/wrapper.rb:152 in 'IO::Endpoint::Wrapper#bind'
| /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/host_endpoint.rb:68 in 'block in IO::Endpoint::HostEndpoint#bind'
| /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/host_endpoint.rb:67 in 'Array#each'
| /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/host_endpoint.rb:67 in 'Enumerator#each'
| /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/host_endpoint.rb:67 in 'Enumerable#map'
| /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/host_endpoint.rb:67 in 'IO::Endpoint::HostEndpoint#bind'
| /opt/fluent/lib/ruby/gems/3.4.0/gems/async-http-0.89.0/lib/async/http/endpoint.rb:216 in 'Async::HTTP::Endpoint#bind'
| /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/generic.rb:82 in 'IO::Endpoint::Generic#accept'
| /opt/fluent/lib/ruby/gems/3.4.0/gems/async-http-0.89.0/lib/async/http/server.rb:67 in 'block in Async::HTTP::Server#run'
| /opt/fluent/lib/ruby/gems/3.4.0/gems/async-2.24.0/lib/async/task.rb:200 in 'block in Async::Task#run'
| /opt/fluent/lib/ruby/gems/3.4.0/gems/async-2.24.0/lib/async/task.rb:438 in 'block in Async::Task#schedule'
```

### Additional context

The crash leaves the `fluent-plugin-prometheus` `/aggregated_metrics` endpoint permanently
returning HTTP 500 (body: `"Connection refused - connect(2) for 127.0.0.1:"`) until
Fluentd is fully restarted. The affected thread is one of the per-worker Prometheus stats
collectors that the aggregation endpoint queries internally.

The stack trace points to `io-endpoint-0.15.2` and `async-http-0.89.0` — the new thread starts
its HTTP server bind before the old thread's socket is fully closed, suggesting either a missing
`SO_REUSEPORT`/`SO_REUSEADDR` option or insufficient drain time before rebinding during a
SIGHUP-triggered reload.

Our workaround is to avoid `systemctl reload` (SIGHUP) in favor of `systemctl restart` (full
stop + start), which guarantees the old process is dead and all sockets released before the new
one starts.

Contributor guide

Open the contributing guide

Research direction

Reproduce the SIGHUP reload with the supplied four-worker configuration, then inspect the worker shutdown and rebind path alongside the io-endpoint-0.15.2 and async-http-0.89.0 stack frames. Verify the fix by confirming no EADDRINUSE warning occurs and /aggregated_metrics continues returning HTTP 200 after reload.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, ruby
Domain
backend, networking, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.