vectordotdev / vectordotdev/vector

Allow headers to be set for all HTTP based sinks

Open
#1,673 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

domain: config domain: networking meta: feedback needs: approval needs: requirements sink: aws_cloudwatch_logs sink: aws_cloudwatch_metrics sink: aws_kinesis_firehose sink: aws_kinesis_streams sink: aws_s3 sink: clickhouse sink: datadog_metrics sink: splunk_hec_logs sink: splunk_hec_metrics
Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

For any HTTP based sink, we should allow users to set custom headers. This is particularly important for services that support special headers (like S3). With S3, you can set all kinds of special headers that set the ACL, encryption mechanism, etc. This is already provided in the http sink and I would like the exact same option available to the following sinks:

  • aws_s3 <---- do this first since users are waiting on this
  • aws_s3 docs addressing how to set ACLs, encryption, etc.
  • aws_cloudwatch_logs
  • aws_cloudwatch_metrics
  • aws_kinesis_firehose
  • aws_kinesis_streams
  • clickhouse?
  • database_metrics
  • gcp_pubsub
  • gcp_stackdriver_logging
  • new_relic_logs
  • sematext
  • splunk_hec

I don't see any downside to providing this option.


For further context, this issue came out of a meeting with @zcapper. They're using the aws_s3 sink to write objects across accounts. This scenario is perfectly described in this AWS tutorial. To summarize, I'll walk through a simple example.

Given two AWS accounts A and B:

  1. Account A is where Vector is deployed.
  2. Account B owns the S3 bucket.
  3. Account A is granted cross-account access to account B's S3 bucket via S3's cross-account bucket permissions.
  4. When account A writes to the bucket, the S3 object ownership remains under account A.
  5. Users in account B cannot modify the object as a result.

This can be easily solved by supplying the x-amz-grant-full-control header when writing the object.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the existing http sink's headers option with the prioritized aws_s3 sink, then inspect the aws_s3 implementation and its documentation. Done means aws_s3 accepts and forwards configurable custom headers, with docs explaining ACL and encryption use; the other listed sinks remain follow-up scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.