Using 'prefixes' with S3 firehose fails to retrieve a single object with only s3:GetObject permissions
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
0.14.0
### Description
The S3 firehose supports the user providing a list of `uris` or `prefixes` to fetch. When using `prefixes`, the implementation issues a 'list objects' call to get the list of objects that match the given prefix. There is a fallback in the implementation where if the request fails due to authorization issues (user does not have `s3:ListBucket` permissions), it will try to retrieve the object directly, using the `s3:GetObject` grant. This makes sense since a typical auth strategy is to grant read permission to specific objects but not allow a user to list out path prefixes. However, the fallback still issues a 'list objects' request which fails if a user has `s3:GetObject` but not `s3:ListBucket` which defeats the purpose of the fallback.
See:
https://github.com/apache/incubator-druid/blob/master/extensions-core/s3-extensions/src/main/java/org/apache/druid/firehose/s3/StaticS3FirehoseFactory.java#L150
https://github.com/apache/incubator-druid/blob/master/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java#L247
The workaround is to use `uris` if you only have `s3:GetObject` permissions, but in the case where you are building an application where you don't know beforehand what permissions the user has, it would be nice to be able to use `prefixes` only and have it work for all cases.
Contributor guide
Research direction
Read StaticS3FirehoseFactory.java around line 150 and S3Utils.java around line 247, following how prefixes handling and the authorization fallback request are connected. Verify the behavior for a prefix when the caller has s3:GetObject but not s3:ListBucket; done means the prefixes-only path works in both permission cases without requiring object URIs in advance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100