vectordotdev / vectordotdev/vector
Add an option for limiting the included time period to `file` sink
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
Vector would benefit from having an option which would limit included time period for a single file in file sink.
It could be similar to batch.timeout_secs in aws_s3 sink, but with important difference that instead of batching the data, Vector would write it directly to the output file, but create a new file if the current file has been opened for more than timeout_secs seconds.
One simple approach is to create new files with different filenames derived from the user-defined template using creation timestamp. In addition, it is possible to support the "rotation" pattern, when the old file is renamed and a new file is created with the same name.
Related to https://github.com/timberio/vector/issues/2057.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the file sink implementation and the related issue #2057. Clarify the intended time-limit behavior and choose between timestamp-derived filenames and rename-based rotation; done means the selected behavior is implemented, documented, and covered by relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100