Downloading a large number of artifacts from S3 is significantly slower than uploading
- Dominant language
- Go
- Stars
- 1.1k
- Forks
- 378
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 74
Description
Presently, `ArtifactDownloader` [creates an S3Downloader](https://github.com/buildkite/agent/blob/57acf17dc5f0c8a1a7a18d8c71600aa4c75f5a19/agent/artifact_downloader.go#L101-L107) (and thus an s3.S3 client) [per artifact](https://github.com/buildkite/agent/blob/57acf17dc5f0c8a1a7a18d8c71600aa4c75f5a19/agent/artifact_downloader.go#L84).
This adds an [`s3:ListObjects`](https://github.com/buildkite/agent/blob/57acf17dc5f0c8a1a7a18d8c71600aa4c75f5a19/agent/s3.go#L138) request in the path of each artifact download as a preflight test for the credentials resulting in higher latency between real download requests.
---
TODO:
- [ ] Examine whether this preflight request is needed or if we can catch the credential errors elsewhere from the real request and apply the same error handling
- [ ] Investigate whether the S3Downloader instance (or at least the underlying s3.S3 client) can be safely shared between multiple download requests
Contributor guide
Research direction
Start with agent/artifact_downloader.go around the ArtifactDownloader and S3Downloader creation, then read agent/s3.go around the s3:ListObjects preflight request. Determine whether credential errors can instead be handled from the real download request and whether an S3Downloader or underlying client can be shared safely. Done means both questions have a documented, tested direction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100