vectordotdev / vectordotdev/vector
Add unique path fingeprinting strategy
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
Context
Currently file source has two fingerprinting strategies, checksum and device_and_inode. Both of them have their advantages and disadvantages. The device_and_inode has the problem of inode reusage and checksum has a problem with log files smaller than fingerprint_bytes which is 256B by default.
Problem
In usecases when inodes are reused and log files can be small, both strategies aren't a good fit. One of the places where such situations arise is on Kubernetes clusters with kubernetes source that is using in the background file source. While using checksum strategy is ok, as we can somewhat mitigate the problem by documenting the pitfall and exposing fingerprint_bytes option, in this case, thanks to Kubernetes CRI specification, we know that log files that we will be reading have unique paths so we can do better.
Solution
Add new unique_path fingerprinting strategy which identifies files by their paths.
The only constraint of this strategy is that users have to guarantee that their log file paths will never be reused.
Occurrences
Our kubernetes tests are logging messages smaller than 256B, so they are currently failing #1949 because we switched from device_and_inode to checksum strategy.
Although that situation is somewhat artificial, a common situation is when the users pod exits quickly so the amount of logs tends to be small. And that situation is even more problematic as it's usually cased by an error, for example: invalid configuration, unknown flag, etc..., which is probably logged to the console but which we won't collect with checksum strategy.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the file source fingerprinting strategies and the Kubernetes source tests mentioned in the issue, then review issue #1949 for the current failures. Implement the unique_path strategy so files are identified by path, and verify that the Kubernetes tests collect log messages smaller than 256B.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100