livepeer / livepeer/go-tools

S3 custom driver doesn't support region parameter

Open
#21 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
0
Forks
1
PR merge metrics
No merged PRs in 30d

Description

[I think we're hardcoding "ignored" for the "region" parameter for `s3+http` and `s3+https` URLs](https://github.com/livepeer/go-tools/blob/9b486a733c2e268a0bb27b124147fdd1ac308f17/drivers/s3.go#L137). This works for most custom S3 storage drivers, but we've encountered one at least that requires it to be set to something:

```
> ./livepeer-catalyst-uploader s3+https://xxx:yyy@s3.eu-central-2.wasabisys.com/bucket/testing.txt
Testing!
time="2023-02-09T21:45:19+01:00" level=fatal msg="AuthorizationHeaderMalformed: The authorization header is malformed; the region 'ignored' is wrong; expecting 'eu-central-2'\n\tstatus code: 400, request id: redacted, host id: redacted"
```

(I strongly suspect that Wasabi storage is using AWS S3 behind the scenes, but whatever.)

Solution 1: Figure out a way to represent the region in the URL somehow. The most obvious way would be a query string `./livepeer-catalyst-uploader s3+https://xxx:yyy@s3.eu-central-2.wasabisys.com/bucket/testing.txt?region=eu-central-2`, but that would be a pain to implement with Mist's directory traversal. Also I kind of like that you can define one of these URLs and concatenate a suffix and get a valid URL.

Solution 2: It occurs to me that we could just teach our driver to be smart enough to fix this on its own... it is being presented with the error
```
AuthorizationHeaderMalformed: The authorization header is malformed; the region 'ignored' is wrong; expecting 'eu-central-2'
```
There's nothing stopping it from just turning around and retying the request with the correct region at that point. Adds one RTT of latency for the upload but we haven't even started streaming any data...

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in drivers/s3.go around line 137, where the region is reportedly set to "ignored". Reproduce the failure with the s3+https Wasabi URL shown in the issue and inspect the uploader entry point. Done means custom S3 endpoints can supply or discover the required region without breaking URL suffix concatenation; the implementation approach still needs agreement.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.