buildkite / buildkite/agent

When using S3 for artifacts, `artifact-agent upload` behaves differently from `artifact_paths` in `pipeline.yaml`

Open
#1,683 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.1k
Forks
378
Avg merge
2d 6h
Merged PRs (30d)
74

Description

I discovered an interesting undocumented feature.

The following configuration works.
```
steps:
- label: ':docker: / :node: - Build'
key: build
plugins:
- ecr#v2.5.0:
login: true
region: 'us-east-1'
- docker#v3.9.0:
image: '.dkr.ecr.us-east-1.amazonaws.com/buildkite/docker/library/node:17'
environment:
- NPM_CONFIG__AUTH
- NPM_CONFIG_REGISTRY
commands:
- 'cd /workdir'
- 'npm install && npm run build'
- 'tar --gzip -cf mock.tar.gz build'
- buildkite-agent artifact upload mock.tar.gz

- label: ':amazon-s3: - Publish'
key: publish
depends_on: build
commands:
- 'buildkite-agent artifact download mock.tar.gz .'
- 'tar xmf mock.tar.gz'
- './scripts/publish.sh build'
plugins:
- cultureamp/aws-assume-role#v0.2.0:
role: arn:aws:iam:::role/cross-acct
```

This configuration does not work.
```
steps:
- label: ':docker: / :node: - Build'
key: build
plugins:
- ecr#v2.5.0:
login: true
region: 'us-east-1'
- docker#v3.9.0:
image: '.dkr.ecr.us-east-1.amazonaws.com/buildkite/docker/library/node:17'
environment:
- NPM_CONFIG__AUTH
- NPM_CONFIG_REGISTRY
commands:
- 'cd /workdir'
- 'npm install && npm run build'
- 'tar --gzip -cf mock.tar.gz build'
artifact_paths:
- mock.tar.gz

- label: ':amazon-s3: - Publish'
key: publish
depends_on: build
commands:
- 'buildkite-agent artifact download mock.tar.gz . --step build'
- 'tar xmf mock.tar.gz'
- './scripts/publish.sh build'
plugins:
- cultureamp/aws-assume-role#v0.2.0:
role: arn:aws:iam:::role/cross-acct
```

Debugging the agent showed me that when using `artifact_paths`, the artifact is retrieved via the S3 client. If the step has a cross-account role without access to the artifact bucket, the step cannot access the artifact.

When creating an artifact use `buildkite-agent artifact upload`, the s3 uploader creates a presigned link which is accessible from the `publish` step using an assumed role.

This concerns me because the functionality is not documented and therefore subject to change.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.