apache / apache/pulsar

Improve handling of S3 offloading configuration in other regions

Open
#9,084 10 comments 1 reaction 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Is your enhancement request related to a problem? Please describe.**
Currently, when using S3 offloading in a region other than us-east-1, the underlying jclouds library does some non-standard AWS things which require both:

A) the policy Pulsar is running as to have `GetBucketLocation` permissions and
B) to change the endpoint to use a region specific endpoint

This causes confusion which is not well documented and difficult to explain, and different from most AWS implementations. See https://github.com/apache/pulsar/issues/3833 for context

**Describe the solution you'd like**
We should do 2 things

1. See if we can eliminate the need for `GetBucketLocation`, looking at https://github.com/apache/jclouds/blob/31a3e5b5df1543d04098e3a694130b7ae8e6e079/apis/s3/src/main/java/org/jclouds/s3/config/S3HttpApiModule.java#L91 it appears to only be used when jclouds detects multiple regions. Where jcloud is getting more than one region from isn't clear, but if the user sets a region, we should just use that single region and skip the getBucketLocationCheck
2. Ensure that setting just the region is sufficient to configure the correct endpoint. Getting rid of the `GetBucketLocation` check may be sufficient such that the default endpoint works, otherwise, we should build the correct endpoint name if the region is specified but no endpoint is manually provided

**Describe alternatives you've considered**
Another consideration (and perhaps still a longer term goal) is to replace the use of jcloud for AWS (but still use it for other cloud providers) as jcloud does have some other behavior that differs from AWS.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by tracing Pulsar's S3 offloading configuration into the jclouds S3 integration, then read the referenced S3HttpApiModule.java logic around region detection and getBucketLocation. Verify behavior for a non-us-east-1 region with no manually supplied endpoint. Done means a configured region avoids the unnecessary GetBucketLocation permission and selects the correct S3 endpoint, with any needed tests passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.