vectordotdev / vectordotdev/vector
Migrate JSON batching sinks to use `JsonArrayBuffer`
Open
Nobody has claimed this yet.
domain: buffers
domain: sinks
sink: aws_cloudwatch_logs
sink: aws_s3
sink: clickhouse
sink: datadog_metrics
sink: elasticsearch
sink: gcp_pubsub
sink: http
sink: splunk_hec_logs
sink: splunk_hec_metrics
type: tech debt
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
Similar to #1659 and related to #1666 we now have a new JsonArrayBuffer type that can be used to unify all the sinks that use arrays of JSON data to send their events:
-
aws_cloudwatch_logs -
aws_s3 -
clickhouse -
datadog_metrics -
elasticsearch -
gcs_pubsub -
gcs_stackdriver_logging -
http -
splunk_hec
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing JsonArrayBuffer implementation and the batching code for the nine listed sinks: aws_cloudwatch_logs, aws_s3, clickhouse, datadog_metrics, elasticsearch, gcs_pubsub, gcs_stackdriver_logging, http, and splunk_hec. Compare each sink with the existing buffer usage. Done means all listed sinks use JsonArrayBuffer while preserving their current event-sending behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- stream-processing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100