buildkite / buildkite/agent

Downloading a large number of artifacts from S3 is significantly slower than uploading

Open
#1,510 0 comments 4 reactions 0 assignees View on GitHub
artifacts performance
Dominant language
Go
Stars
1.1k
Forks
378
Avg merge
2d 6h
Merged PRs (30d)
74

Description

Presently, `ArtifactDownloader` [creates an S3Downloader](https://github.com/buildkite/agent/blob/57acf17dc5f0c8a1a7a18d8c71600aa4c75f5a19/agent/artifact_downloader.go#L101-L107) (and thus an s3.S3 client) [per artifact](https://github.com/buildkite/agent/blob/57acf17dc5f0c8a1a7a18d8c71600aa4c75f5a19/agent/artifact_downloader.go#L84).

This adds an [`s3:ListObjects`](https://github.com/buildkite/agent/blob/57acf17dc5f0c8a1a7a18d8c71600aa4c75f5a19/agent/s3.go#L138) request in the path of each artifact download as a preflight test for the credentials resulting in higher latency between real download requests.

---

TODO:

- [ ] Examine whether this preflight request is needed or if we can catch the credential errors elsewhere from the real request and apply the same error handling
- [ ] Investigate whether the S3Downloader instance (or at least the underlying s3.S3 client) can be safely shared between multiple download requests

Contributor guide

Open the contributing guide

Research direction

Start with agent/artifact_downloader.go around the ArtifactDownloader and S3Downloader creation, then read agent/s3.go around the s3:ListObjects preflight request. Determine whether credential errors can instead be handled from the real download request and whether an S3Downloader or underlying client can be shared safely. Done means both questions have a documented, tested direction.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.