vectordotdev / vectordotdev/vector

AWS S3 sink healthchecks require permissive ACLs

Open
#620 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

domain: healthchecks have: should provider: aws sink: aws_s3
Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

For the aws_s3 sink, writes to the S3 bucket only require file-level s3:PutObject permissions, however healthchecks are done via HeadBucket which requires the bucket-level s3:ListBucket permissions.

While I'm sure this is fine for some use cases, it does mean giving writers access to view the entire bucket's contents, even if they are only writing to a specific prefix, which seems too permissive given it's just for healthchecks.
The value of a healthcheck here is also questionable given that it doesn't simulate the permissions/API call required to actually store a log file.

I would suggest using the same approach as AWS ALB access logging of writing a test file to
{{ bucket_name }}/{{ key_prefix }}/AccessLogTestFile

Or alternatively giving an option to disable/ignore healthchecks.

In my use-case where there are multiple AWS accounts writing to a single logging bucket, giving s3:ListBucket would mean allowing any account to list log files written by other accounts (as well as other hosts in its own account), whereas for example, the preferred ACL for my use-case would be:

  EventsPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: ...
      PolicyDocument:
        Version: 2012-10-17

        Statement:
          - Sid: ChildAccountDeliveryForAcc90000000000
            Effect: Allow
            Principal:
              AWS: 90000000000
            Action: s3:PutObject
            Resource: arn:aws:s3:::logging-bucket/AppLogs/90000000000/*

          - Sid: ChildAccountDeliveryForAcc91111111111
            Effect: Allow
            Principal:
              AWS: 91111111111
            Action: s3:PutObject
            Resource: arn:aws:s3:::logging-bucket/AppLogs/91111111111/*

which can be achieved with the key_prefix setting, assuming heathchecks aren't required/enforced.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the aws_s3 sink healthcheck implementation and inspect how HeadBucket and key_prefix are used. Compare the healthcheck permissions with the permissions required for writing a log file, then determine whether a write-based check or an option to disable healthchecks best fits the stated use case. Done means the sink no longer requires unnecessary bucket listing access while preserving clear healthcheck behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.