[Meta] Reduce memory/goroutines per idle filestream receivers
@blakerouse is already working on this.
Since Jul 31, 2026.
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 370
Description
Why
When Filebeat runs under elastic-agent as an OTel receiver, agent creates one filebeatreceiver per input stream (elastic-agent/internal/pkg/otel/translate/otelconfig.go:389,488-504). Each receiver is a full beat.Beat: its own logger, monitoring registries, publisher pipeline, input manager, registry view, prospector and harvester scheduler, even though a receiver typically watches one file or one directory.
Almost none of the machinery that duplicates per receiver is doing work. It sits blocked: goroutines parked on channels nothing writes to, cleanup loops for input types the receiver never instantiates, registry scans for state that never arrives. A deployment with 50 receivers ran 1,204 goroutines to tail 50 files.
Two things scale badly:
- Idle goroutines — a fixed ~24 per receiver, all but a couple parked.
- Registry memory, quadratically —
readStatesloads every key carrying the input type's prefix, not just the keys belonging to the input doing the loading. Agent points every receiver of a component at the samepath.data, so on any restart each receiver loads every other receiver's state.
This issue records the measured effect of fixing both, and proposes ways to split up the work into different changes.
Measured impact
All figures from x-pack/filebeat/fbreceiver, Apple M2 Pro, -count=1, withidentical benchmark code on both sides (the benchmarks themselves are the first deliverable, see PR 1).
Goroutines: idle fleet, one file per receiver
BenchmarkNReceiverFootprint: N receivers, one filestream input each, sharing one path.data; measured after full ingestion, once idle.
| receivers | before (total) | after (total) | before per recv | after [er recv |
|---|---|---|---|---|
| 1 | 28 | 11 | 24.0 | 7.0 |
| 10 | 244 | 29 | 24.0 | 2.5 |
| 50 | 1,204 | 109 | 24.0 | 2.1 |
−91% at 50 receivers. The per-receiver figure converges to ~2.1 because the shared components amortise; the marginal cost of the 51st receiver is 2 goroutines, not 24.
Extrapolated to 1,000 receivers: ~24,000 → ~2,100.
Memory — registry state on restart
BenchmarkNReceiverRegistryFootprint: 300 files per receiver, fleet restarted against a warm registry (the state a long-running deployment is always in).
| receivers | heap-B/file before | after | heap-B/receiver before | after |
|---|---|---|---|---|
| 2 | 7,289 | (0, GC-clamped) | 2.19 MB | (0, GC-clamped) |
| 10 | 16,830 | 8,106 | 5.05 MB | 2.43 MB |
−52% per receiver at N=10, and the shape matters more than the point value: per-receiver memory before grows with the number of co-tenants (7,289 → 16,830 B/file going 2 → 10 receivers, +131%), because each receiver holds a copy of every other's state. After, it is flat in N.
Total for that fleet: 50.5 MB → 24.3 MB. The gap widens with receiver count.
Memory — idle fleet
| receivers | heap-B per receiver before | after per receiver |
|---|---|---|
| 1 | 48,832 | (0, GC-clamped) |
| 10 | 93,292 | 64,956 |
| 50 | 164,066 | 95,821 |
−42% at 50 receivers. Note the before column also grows with N; the after
column grows far more slowly.
Where the 24 exist
| per receiver | component |
|---|---|
| 9 | input-cursor registry cleaners, for input types never configured |
| 4 | publisher pipeline: queueReader, eventConsumer, spawningWorker, reapClosedClients |
| 3 | beater lifecycle: signalWait ×2, done→context bridge |
| 3 | filestream watcher: scan loop, notify drain, FS-event loop |
| 2 | filestream: harvester waker, registry update writer |
| 1 | filestream registry cleaner |
| 1 | legacy registrar, idle unless a log-family input runs |
| 1 | runScans - filestream's scan loop, doing actual work per receiver |
| 1 | beater.Run - parked; needs a beat.Beater contract change |
Remove 22, keep 2
22 of the goroutines above can be completely removed. 2 most likely should remain:
runScans- removing it means makingv2.Input.Runnon-blocking (~15 implementations) and building a shared scan scheduler. Neither helps alone: with a blockingRunthe goroutine exists regardless, and with a non-blockingRunbut no scheduler filestream just spawns its own ticker.beater.Run-beat.Beater.Runblocks for the Beat's lifetime. Changing that touches every Beat.
How to reach this goal
- #52364
- #52350
- #52351
- #52352
- #52353
- #52354
- #52355
- #52356
- #52357
- #52358
- #52359
- #52360
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.