apache / apache/druid

Using 'prefixes' with S3 firehose fails to retrieve a single object with only s3:GetObject permissions

Open
#7,573 1 comment 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

0.14.0

### Description

The S3 firehose supports the user providing a list of `uris` or `prefixes` to fetch. When using `prefixes`, the implementation issues a 'list objects' call to get the list of objects that match the given prefix. There is a fallback in the implementation where if the request fails due to authorization issues (user does not have `s3:ListBucket` permissions), it will try to retrieve the object directly, using the `s3:GetObject` grant. This makes sense since a typical auth strategy is to grant read permission to specific objects but not allow a user to list out path prefixes. However, the fallback still issues a 'list objects' request which fails if a user has `s3:GetObject` but not `s3:ListBucket` which defeats the purpose of the fallback.

See:
https://github.com/apache/incubator-druid/blob/master/extensions-core/s3-extensions/src/main/java/org/apache/druid/firehose/s3/StaticS3FirehoseFactory.java#L150

https://github.com/apache/incubator-druid/blob/master/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java#L247

The workaround is to use `uris` if you only have `s3:GetObject` permissions, but in the case where you are building an application where you don't know beforehand what permissions the user has, it would be nice to be able to use `prefixes` only and have it work for all cases.

Contributor guide

Open the contributing guide

Research direction

Read StaticS3FirehoseFactory.java around line 150 and S3Utils.java around line 247, following how prefixes handling and the authorization fallback request are connected. Verify the behavior for a prefix when the caller has s3:GetObject but not s3:ListBucket; done means the prefixes-only path works in both permission cases without requiring object URIs in advance.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.