S3 custom driver doesn't support region parameter
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 0
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
[I think we're hardcoding "ignored" for the "region" parameter for `s3+http` and `s3+https` URLs](https://github.com/livepeer/go-tools/blob/9b486a733c2e268a0bb27b124147fdd1ac308f17/drivers/s3.go#L137). This works for most custom S3 storage drivers, but we've encountered one at least that requires it to be set to something:
```
> ./livepeer-catalyst-uploader s3+https://xxx:yyy@s3.eu-central-2.wasabisys.com/bucket/testing.txt
Testing!
time="2023-02-09T21:45:19+01:00" level=fatal msg="AuthorizationHeaderMalformed: The authorization header is malformed; the region 'ignored' is wrong; expecting 'eu-central-2'\n\tstatus code: 400, request id: redacted, host id: redacted"
```
(I strongly suspect that Wasabi storage is using AWS S3 behind the scenes, but whatever.)
Solution 1: Figure out a way to represent the region in the URL somehow. The most obvious way would be a query string `./livepeer-catalyst-uploader s3+https://xxx:yyy@s3.eu-central-2.wasabisys.com/bucket/testing.txt?region=eu-central-2`, but that would be a pain to implement with Mist's directory traversal. Also I kind of like that you can define one of these URLs and concatenate a suffix and get a valid URL.
Solution 2: It occurs to me that we could just teach our driver to be smart enough to fix this on its own... it is being presented with the error
```
AuthorizationHeaderMalformed: The authorization header is malformed; the region 'ignored' is wrong; expecting 'eu-central-2'
```
There's nothing stopping it from just turning around and retying the request with the correct region at that point. Adds one RTT of latency for the upload but we haven't even started streaming any data...
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in drivers/s3.go around line 137, where the region is reportedly set to "ignored". Reproduce the failure with the s3+https Wasabi URL shown in the issue and inspect the uploader entry point. Done means custom S3 endpoints can supply or discover the required region without breaking URL suffix concatenation; the implementation approach still needs agreement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100