elastic / elastic/beats

[Meta] Reduce memory/goroutines per idle filestream receivers

Open
#52,349 3 comments 4 reactions 1 assignee View on GitHub

@blakerouse is already working on this.

Since Jul 31, 2026.

Team:Elastic-Agent-Control-Plane Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

Why

When Filebeat runs under elastic-agent as an OTel receiver, agent creates one filebeatreceiver per input stream (elastic-agent/internal/pkg/otel/translate/otelconfig.go:389,488-504). Each receiver is a full beat.Beat: its own logger, monitoring registries, publisher pipeline, input manager, registry view, prospector and harvester scheduler, even though a receiver typically watches one file or one directory.

Almost none of the machinery that duplicates per receiver is doing work. It sits blocked: goroutines parked on channels nothing writes to, cleanup loops for input types the receiver never instantiates, registry scans for state that never arrives. A deployment with 50 receivers ran 1,204 goroutines to tail 50 files.

Two things scale badly:

  1. Idle goroutines — a fixed ~24 per receiver, all but a couple parked.
  2. Registry memory, quadraticallyreadStates loads every key carrying the input type's prefix, not just the keys belonging to the input doing the loading. Agent points every receiver of a component at the same path.data, so on any restart each receiver loads every other receiver's state.

This issue records the measured effect of fixing both, and proposes ways to split up the work into different changes.

Measured impact

All figures from x-pack/filebeat/fbreceiver, Apple M2 Pro, -count=1, withidentical benchmark code on both sides (the benchmarks themselves are the first deliverable, see PR 1).

Goroutines: idle fleet, one file per receiver

BenchmarkNReceiverFootprint: N receivers, one filestream input each, sharing one path.data; measured after full ingestion, once idle.

receivers before (total) after (total) before per recv after [er recv
1 28 11 24.0 7.0
10 244 29 24.0 2.5
50 1,204 109 24.0 2.1

−91% at 50 receivers. The per-receiver figure converges to ~2.1 because the shared components amortise; the marginal cost of the 51st receiver is 2 goroutines, not 24.

Extrapolated to 1,000 receivers: ~24,000 → ~2,100.

Memory — registry state on restart

BenchmarkNReceiverRegistryFootprint: 300 files per receiver, fleet restarted against a warm registry (the state a long-running deployment is always in).

receivers heap-B/file before after heap-B/receiver before after
2 7,289 (0, GC-clamped) 2.19 MB (0, GC-clamped)
10 16,830 8,106 5.05 MB 2.43 MB

−52% per receiver at N=10, and the shape matters more than the point value: per-receiver memory before grows with the number of co-tenants (7,289 → 16,830 B/file going 2 → 10 receivers, +131%), because each receiver holds a copy of every other's state. After, it is flat in N.

Total for that fleet: 50.5 MB → 24.3 MB. The gap widens with receiver count.

Memory — idle fleet
receivers heap-B per receiver before after per receiver
1 48,832 (0, GC-clamped)
10 93,292 64,956
50 164,066 95,821

−42% at 50 receivers. Note the before column also grows with N; the after
column grows far more slowly.

Where the 24 exist
per receiver component
9 input-cursor registry cleaners, for input types never configured
4 publisher pipeline: queueReader, eventConsumer, spawningWorker, reapClosedClients
3 beater lifecycle: signalWait ×2, done→context bridge
3 filestream watcher: scan loop, notify drain, FS-event loop
2 filestream: harvester waker, registry update writer
1 filestream registry cleaner
1 legacy registrar, idle unless a log-family input runs
1 runScans - filestream's scan loop, doing actual work per receiver
1 beater.Run - parked; needs a beat.Beater contract change

Remove 22, keep 2

22 of the goroutines above can be completely removed. 2 most likely should remain:

  • runScans - removing it means making v2.Input.Run non-blocking (~15 implementations) and building a shared scan scheduler. Neither helps alone: with a blocking Run the goroutine exists regardless, and with a non-blocking Run but no scheduler filestream just spawns its own ticker.
  • beater.Run - beat.Beater.Run blocks for the Beat's lifetime. Changing that touches every Beat.

How to reach this goal

  1. #52364
  2. #52350
  3. #52351
  4. #52352
  5. #52353
  6. #52354
  7. #52355
  8. #52356
  9. #52357
  10. #52358
  11. #52359
  12. #52360

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.