vectordotdev / vectordotdev/vector
Only warn about small files when they are not empty
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
This is a follow up to https://github.com/timberio/vector/pull/863#issuecomment-544314174.
To recap, when the file source is configured to use the checksum fingerprinting strategy a warning log message is emitted letting the user know the file is too small to generate a fingerprint:
warn!(message = "Ignoring file smaller than fingerprint_bytes", file = ?path);
This is a very welcome feature for files that are actually too small. It helps to prevent confusion around why small files aren't being tailed. Unfortunately, it can produce a lot of noise in production environments where many empty files are present.
I'm unsure if we should ignore empty files, given that it could lead to the same confusion that we experienced previously with small files. Then again, I can see the reasoning behind ignoring empty files. I'm opening this up for discussion so we can come to a conclusion about what to do in this scenario.
Update:
New log line for too small files is:
Currently ignoring file too small to fingerprint
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the file source's checksum fingerprinting path and the warning that reports files as too small to fingerprint. The issue does not name a file or test and leaves the desired handling of empty files unresolved, so first confirm the intended behavior before identifying the relevant coverage and defining completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100