filestream: let the prospector own the scan loop
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
## Why
Watching a file system costs three goroutines per input:
1. the file watcher's scan loop (plus a `time.Tick` that is never stopped),
2. a goroutine draining harvester-closed notifications off a buffered channel,
3. the prospector's FS-event consumer, reading an unbuffered channel.
Meanwhile the goroutine that the input framework already provides, the one running `Input.Run` sits parked doing nothing but waiting for cancellation.
## What to change
Replace `FSWatcher.Run` + `Event()` + the unbuffered event channel with `ScanOnce(ctx, sink, ...)`, which hands each event straight to the prospector. Replace `SetObserver(chan HarvesterStatus)` with a `HarvesterObserver` interface called directly, the channel needed a drain goroutine only because a scan can
run for minutes and a closing harvester must not block on it.
The scan loop then runs on the goroutine already executing the input. Three goroutines become none, and events no longer make a channel round-trip.
A non-positive `check_interval` still disables re-scanning, matching what `time.Tick` did here.
## Why not a shared scan scheduler
It would save nothing. `compat/compat.go:138` spawns a goroutine per input to call `Input.Run`, which must block for the input's lifetime. A shared scheduler would leave that goroutine parked instead of working, while adding a semaphore and cross-input head-of-line blocking (a slow scan, or a blocking `hg.Stop`,
occupying a shared worker).
Contributor guide
Research direction
Start at compat/compat.go:138 to understand the input goroutine, then trace FSWatcher.Run, Event(), SetObserver, and the prospector's Input.Run path. Replace the channel-based flow with ScanOnce(ctx, sink, ...) and direct HarvesterObserver calls, preserving disabled re-scanning for non-positive check_interval. Done means the scan runs on the input goroutine with no extra watcher, drain, or event-consumer goroutines.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100