[Filestream] Files can be re-ingested on start up because of `clean_removed: true` (that's the default)
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 370
Description
When Filebeat is restarted it can re-ingest files if hey have been rotated and the rotated paths are also monitored by Filestream.
Given the following Filestream configuration:
```yaml
filebeat.inputs:
- type: filestream
id: oops-I-re-ingested-a-file
paths:
- /tmp/*.log
output.console:
enabled: true
pretty: true
```
And a file `/tmp/foo.log` with some data.
1. Create a file with some data `docker run -it --rm mingrammer/flog -n 2 > /tmp/flog.log`
1. Start Filebeat with the configuration above
2. Wait until the file is fully ingested (no more events on the console)
3. Stop Filebeat
4. Move `/tmp/foo.log` to `/tmp/foo-1.log`: `mv /tmp/flog.log /tmp/flog-1.log`
5. Start Filebeat
Once Filebeat is restarted the file is re-ingested.
However if after moving `/tmp/foo.log` to `/tmp/foo-1.log` a new `/tmp/foo.log` is created (the contents do not matter), like on a common log rotation strategy, no data is duplicated.
The actual issue comes from how the store clean up is implemented:
https://github.com/elastic/beats/blob/5449535b768a9308714a63dc745911c924da307b/filebeat/input/filestream/prospector.go#L89-L101
It checks if `meta.source` from the registry entry matches any of the current files discovered by the `filewatcher` if they do not match, then the entry is removed.
Contributor guide
Research direction
Reproduce the restart scenario with the Filestream configuration and then read filebeat/input/filestream/prospector.go around lines 89-101, where registry entries are cleaned against files discovered by the filewatcher. Confirm the behavior with an appropriate Filestream test: moving a monitored file without creating its replacement must not cause previously ingested data to be emitted again, while normal rotation remains unaffected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100