sameersbn / sameersbn/docker-gitlab

Cannot use S3 with GITLAB_ARTIFACTS_OBJECT_STORE

Open
#2,761 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
8.1k
Forks
2.1k
Avg merge
17h 47m
Merged PRs (30d)
7

Description

There seem to be some issue with GITLAB_ARTIFACTS_OBJECT_STORE_* parameters. It seems to not be working correctly with S3 URLs.

We're using the following parameters

      - GITLAB_ARTIFACTS_OBJECT_STORE_ENABLED=true
      - GITLAB_ARTIFACTS_OBJECT_STORE_REMOTE_DIRECTORY="sample-bucket-name"
      - GITLAB_ARTIFACTS_OBJECT_STORE_DIRECT_UPLOAD=false
      - GITLAB_ARTIFACTS_OBJECT_STORE_PROXY_DOWNLOAD=true
      - GITLAB_ARTIFACTS_OBJECT_STORE_BACKGROUND_UPLOAD=true
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_PATH_STYLE=false
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_PROVIDER=AWS
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_ACCESS_KEY_ID=******
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_SECRET_ACCESS_KEY=******
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_REGION=us-east-1
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_HOST="s3.us-east-1.amazonaws.com"

In this case when trying to run rake gitlab:artifacts:migrate we'll get
bad URI(is not URI?): "://sample-bucket-name.s3.amazonaws.com:"

This URL doesn't seem correct since it's missing a region, it's even more interesting because we specified GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_HOST and in theory it should use it when forming URL, but it does not. We tried removing double quotes around GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_HOST, we tried not using GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_HOST at all, we tried using GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_PATH_STYLE=true instead of false, none of these seem to give correct URI for the S3.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing rake gitlab:artifacts:migrate with the listed GITLAB_ARTIFACTS_OBJECT_STORE settings. Trace how the artifact object-store configuration constructs the S3 URL, especially the host, region, and path-style options. Done means migration forms a valid S3 URI and completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.