elastic / elastic/beats

[Filestream] Files can be re-ingested on start up because of `clean_removed: true` (that's the default)

Open
#43,649 3 comments 0 reactions 0 assignees View on GitHub
bug Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

When Filebeat is restarted it can re-ingest files if hey have been rotated and the rotated paths are also monitored by Filestream.

Given the following Filestream configuration:
```yaml
filebeat.inputs:
- type: filestream
id: oops-I-re-ingested-a-file
paths:
- /tmp/*.log

output.console:
enabled: true
pretty: true
```
And a file `/tmp/foo.log` with some data.

1. Create a file with some data `docker run -it --rm mingrammer/flog -n 2 > /tmp/flog.log`
1. Start Filebeat with the configuration above
2. Wait until the file is fully ingested (no more events on the console)
3. Stop Filebeat
4. Move `/tmp/foo.log` to `/tmp/foo-1.log`: `mv /tmp/flog.log /tmp/flog-1.log`
5. Start Filebeat

Once Filebeat is restarted the file is re-ingested.

However if after moving `/tmp/foo.log` to `/tmp/foo-1.log` a new `/tmp/foo.log` is created (the contents do not matter), like on a common log rotation strategy, no data is duplicated.

The actual issue comes from how the store clean up is implemented:
https://github.com/elastic/beats/blob/5449535b768a9308714a63dc745911c924da307b/filebeat/input/filestream/prospector.go#L89-L101

It checks if `meta.source` from the registry entry matches any of the current files discovered by the `filewatcher` if they do not match, then the entry is removed.

Contributor guide

Open the contributing guide

Research direction

Reproduce the restart scenario with the Filestream configuration and then read filebeat/input/filestream/prospector.go around lines 89-101, where registry entries are cleaned against files discovered by the filewatcher. Confirm the behavior with an appropriate Filestream test: moving a monitored file without creating its replacement must not cause previously ingested data to be emitted again, while normal rotation remains unaffected.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.