Ability to "register" a file previously uploaded as an artifact
- Dominant language
- Go
- Stars
- 1.1k
- Forks
- 378
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 74
Description
If I use `buildkite-agent artifact upload *.gz s3://my-bucket/cache` (such that the upload location is shared among multiple builds), the buildkite agent will always upload everything. While this is a good default, it would be nice to support skipping things that have already been uploaded previously (e.g. files that already exist).
For these more complex workflows, it would be quite useful to be able to do something like `buildkite-agent artifact register ` where `register` does not upload the file, but instead creates the metadata entries that allow `buildkite-agent artifact download` and list the files in the webUI's artifacts list. Alternatively, `buildkite-agent artifact upload --dont-actually-upload` might work, but I think it's slightly less elegant. ;)
Without this, plugins and workflows that want to perform more complex uploading/downloading must either live with inefficiencies or do without the beautiful artifact webui links in the build pages:
* Example Scenario: A memoization plugin such as [coppermind](https://github.com/staticfloat/coppermind-buildkite-plugin), is used in a large monorepo that contains training scripts for ML models. The memoization script is used to avoid running expensive steps when the inputs have not changed, and it uses the `buildkite-agent artifacts upload` command to store the outputs in a content-addressed manner. Uploading the results of a build step is easy enough, but when that step gets re-run and the memoization plugin realizes it does not have to run the command, it is a large UX degradation that the artifacts have no way of being listed in that step for user examination. Currently, in our workflows, we download the artifacts and upload them again, overwriting them on our S3 bucket, just so that those links can show up in the webUI.
* Example Scenario: A daily job runs that collects logs from servers, but because of the `logrotate` configuration on the servers, it is always collecting the last 30 days worth. We can use the `buildkite-agent artifact upload` command to upload logs into a shared S3 prefix, but it will always upload everything; there's no way to skip things that have already been uploaded. If we use an external tool to upload the files (such as `aws s3 sync ${LOGS_DIR} s3://bucket/prefix`) we can avoid the extra uploading work, but we lose the nice webUI links.
As an aside, I know that `buildkite-agent artifact upload` wasn't really designed for working in a shared prefix; it kind of assumes that it is uploading into a build-job-specific zone, and as such doesn't have support for things like testing if an artifact already exists, skipping the upload if an artifact already exists, etc... but these features would be pretty nice, as the glob matching behavior and ubiquity of `buildkite-agent` make it a nice tool to use in plugins.
Thanks for all of your hard work!
Contributor guide
Research direction
Start by tracing the artifact upload and download/list entry points named in the issue, including the proposed `artifact register` command. Determine how artifact metadata is created and exposed to the web UI, then define the behavior needed for registration without uploading. Done means registered files can be downloaded and listed like uploaded artifacts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100