vectordotdev / vectordotdev/vector
Allow headers to be set for all HTTP based sinks
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
For any HTTP based sink, we should allow users to set custom headers. This is particularly important for services that support special headers (like S3). With S3, you can set all kinds of special headers that set the ACL, encryption mechanism, etc. This is already provided in the http sink and I would like the exact same option available to the following sinks:
-
aws_s3<---- do this first since users are waiting on this -
aws_s3docs addressing how to set ACLs, encryption, etc. -
aws_cloudwatch_logs -
aws_cloudwatch_metrics -
aws_kinesis_firehose -
aws_kinesis_streams -
clickhouse? -
database_metrics -
gcp_pubsub -
gcp_stackdriver_logging -
new_relic_logs -
sematext -
splunk_hec
I don't see any downside to providing this option.
For further context, this issue came out of a meeting with @zcapper. They're using the aws_s3 sink to write objects across accounts. This scenario is perfectly described in this AWS tutorial. To summarize, I'll walk through a simple example.
Given two AWS accounts A and B:
- Account
Ais where Vector is deployed. - Account
Bowns the S3 bucket. - Account
Ais granted cross-account access to accountB's S3 bucket via S3's cross-account bucket permissions. - When account
Awrites to the bucket, the S3 object ownership remains under accountA. - Users in account
Bcannot modify the object as a result.
This can be easily solved by supplying the x-amz-grant-full-control header when writing the object.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the existing http sink's headers option with the prioritized aws_s3 sink, then inspect the aws_s3 implementation and its documentation. Done means aws_s3 accepts and forwards configurable custom headers, with docs explaining ACL and encryption use; the other listed sinks remain follow-up scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100