elastic / elastic/beats

filestream: let the prospector own the scan loop

Open
#52,352 1 comment 1 reaction 0 assignees View on GitHub
Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

## Why

Watching a file system costs three goroutines per input:

1. the file watcher's scan loop (plus a `time.Tick` that is never stopped),
2. a goroutine draining harvester-closed notifications off a buffered channel,
3. the prospector's FS-event consumer, reading an unbuffered channel.

Meanwhile the goroutine that the input framework already provides, the one running `Input.Run` sits parked doing nothing but waiting for cancellation.

## What to change

Replace `FSWatcher.Run` + `Event()` + the unbuffered event channel with `ScanOnce(ctx, sink, ...)`, which hands each event straight to the prospector. Replace `SetObserver(chan HarvesterStatus)` with a `HarvesterObserver` interface called directly, the channel needed a drain goroutine only because a scan can
run for minutes and a closing harvester must not block on it.

The scan loop then runs on the goroutine already executing the input. Three goroutines become none, and events no longer make a channel round-trip.

A non-positive `check_interval` still disables re-scanning, matching what `time.Tick` did here.

## Why not a shared scan scheduler

It would save nothing. `compat/compat.go:138` spawns a goroutine per input to call `Input.Run`, which must block for the input's lifetime. A shared scheduler would leave that goroutine parked instead of working, while adding a semaphore and cross-input head-of-line blocking (a slow scan, or a blocking `hg.Stop`,
occupying a shared worker).

Contributor guide

Open the contributing guide

Research direction

Start at compat/compat.go:138 to understand the input goroutine, then trace FSWatcher.Run, Event(), SetObserver, and the prospector's Input.Run path. Replace the channel-based flow with ScanOnce(ctx, sink, ...) and direct HarvesterObserver calls, preserving disabled re-scanning for non-positive check_interval. Done means the scan runs on the input goroutine with no extra watcher, drain, or event-consumer goroutines.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.