Filestream's `read_until_eof` blocks configuration reloading
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 364
Description
When `read_until_eof` is enabled (default since 9.5.0, backported to 9.3.8/9.4.4) and a filestream stream is removed from config, [`StopHarvesters`](https://github.com/elastic/beats/blob/29f425409c1b60c09d37ad6becbb2abdc1d639e9/filebeat/input/filestream/internal/input-logfile/harvester_runner.go#L953) calls [`drainAndStop`](https://github.com/elastic/beats/blob/29f425409c1b60c09d37ad6becbb2abdc1d639e9/filebeat/input/filestream/internal/input-logfile/harvester_runner.go#L995), which can hold for up to `readUntilEOF.Timeout` (default 1m) + [`stuckGrace`](https://github.com/elastic/beats/blob/29f425409c1b60c09d37ad6becbb2abdc1d639e9/filebeat/input/filestream/internal/input-logfile/harvester_runner.go#L1082) (1m) = 2 minutes before returning.
This causes trouble when trying to reload the configuration and stop the stream, although with different effects depending on path.
## Filebeat standalone and elastic-agent
[`RunnerList.Reload`](https://github.com/elastic/beats/blob/29f425409c1b60c09d37ad6becbb2abdc1d639e9/libbeat/cfgfile/list.go#L132) stops removed runners and calls `wg.Wait()` before starting any new runners, while holding its mutex throughout. [`compat.runner.Stop`](https://github.com/elastic/beats/blob/29f425409c1b60c09d37ad6becbb2abdc1d639e9/filebeat/input/v2/compat/compat.go#L173) has no timeout: it cancels the input context and then blocks on `wg.Wait()` until `Input.Run` returns. `StopHarvesters` is called from inside `Input.Run` (via the prospector's deferred `stopHarvesterGroup`), so the entire reload blocks for the drain duration. Any new streams added in the same config change do not start until the drain completes.
Under elastic-agent, this is worse. [`unitListen`](https://github.com/elastic/beats/blob/29f425409c1b60c09d37ad6becbb2abdc1d639e9/x-pack/libbeat/management/managerV2.go#L534) is a single goroutine; it calls [`cm.reload`](https://github.com/elastic/beats/blob/29f425409c1b60c09d37ad6becbb2abdc1d639e9/x-pack/libbeat/management/managerV2.go#L600) synchronously from the debounce timer case, blocking the goroutine for the full drain. The elastic-agent-client's `changesCh` is unbuffered (`client_v2.go:314`); the gRPC checkin reader goroutine (`applyExpected`) blocks on `changesCh <-` as soon as `unitListen` is occupied and the next `CheckinExpected` arrives.
The reader holds `unitsMu` for write while sending on `changesCh` (`client_v2.go:653`). The checkin writer goroutine calls `sendObserved` which needs `unitsMu.RLock`. So if any genuine `CheckinExpected` change arrives while the drain is running:
1. Reader blocks on `changesCh <-` while holding `unitsMu` write-lock.
2. Writer blocks on `unitsMu.RLock`, observed checkins stop.
3. Agent's command runtime ticks every 30s; at `maxCheckinMisses = 3` it kills the process.
4. Threshold ≈ 90s. Default worst-case drain = 2m.
The result is the process is SIGKILLed mid-drain - the exact outcome `read_until_eof` was introduced to prevent - with no registry flush, so every stream in the filebeat restarts from its last acked offset.
If no concurrent change arrives, beats keeps checking in (writer is independent), the watchdog is quiet, and the only observable effect is that Fleet reports the policy as applied and healthy while the removed stream is still ingesting for up to 2m.
---
## Otel receiver Shutdown
[`filebeatReceiver.Shutdown`](https://github.com/elastic/beats/blob/29f425409c1b60c09d37ad6becbb2abdc1d639e9/x-pack/filebeat/fbreceiver/receiver.go#L33) delegates to [`BeatReceiver.Shutdown`](https://github.com/elastic/beats/blob/29f425409c1b60c09d37ad6becbb2abdc1d639e9/x-pack/libbeat/cmd/instance/receiver.go#L216), which calls `Filebeat.StopWithContext` (closes `done`), then waits on `<-br.runDone` bounded by the OTel context deadline. `Run`'s shutdown sequence calls `inputs.Stop()` -> `RunnerList.Stop()` -> each `runner.Stop()` + `wg.Wait()` - none of these inner waits are context-bounded. The OTel deadline can expire and `BeatReceiver.Shutdown` returns, but the drain goroutines inside the process continue running, holding file descriptors and consuming queue space until `stuckGrace` fires or output drains.
For a full component stop (not just stream removal), this is compounded by `drainAndStop` running on every active filestream input concurrently, each with its own 1m budget. The OTel collector's shutdown timeout governs only the outer wait, not the actual work.
With partial config reload enabled, this basically blocks all new config changes until the runner is done.
Contributor guide
Research direction
Start with StopHarvesters and drainAndStop in filebeat/input/filestream/internal/input-logfile/harvester_runner.go, then trace RunnerList.Reload, compat.runner.Stop, and the elastic-agent unitListen path named in the report. Reproduce removal and full-stop behavior with read_until_eof enabled, including concurrent agent check-ins and OTel shutdown. Done means reload and shutdown do not remain blocked by an unbounded inner drain or trigger the described check-in/watchdog failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100