vectordotdev / vectordotdev/vector
New `aws_kinesis_streams` source
Open
Nobody has claimed this yet.
domain: logs
domain: sources
have: should
needs: approval
needs: requirements
provider: aws
source: new
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
It would be nice if Vector could ingest logs from a AWS Kinesis data stream (not Firehose which is covered in #3566).
Requirements
- Ability to exclusively read partitions across multiple Vector instances.
- Checkpointing the stream to resume properly when Vector is restarted (and prevent data loss).
- Bonus points if checkpoints can be stored remotely, like in Dynamo.
- Ability to specify where to start reading from the stream (horizon, etc).
- Add the
kinesis.streamandkinesis.partitionas context fields (.denoting nested fields). - Ability to merge split/multi-line messages.
- It should handle all of the various stream statuses and react accordingly.
- Consider using the new enhanced fanout and HTTP2 capabilities to avoid polling.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the requirements in this issue and the AWS StreamDescription API documentation, then compare the existing Firehose work referenced in #3566. Done means Vector can ingest Kinesis streams with partition ownership, restart-safe checkpointing, configurable starting positions, context fields, multiline handling, and appropriate stream-status behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- backend, cloud, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100