buildkite / buildkite/agent

Support flattening lists in step plugins list

Open
#1,634 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.1k
Forks
378
Avg merge
2d 6h
Merged PRs (30d)
74

Description

**Is your feature request related to a problem? Please describe.**
In an effort to keep our pipeline steps more concise we're using YAML anchors and aliases to define the plugins for our steps at the top level and we reuse the lists in multiple places. This works well but we have some steps that need to fetch artifacts or add additional plugins. Since YAML doesn't support merging lists with anchors like it does for dictionaries, we need to redefine the list of plugins for these other use cases. Because of how our caches are layered, our lists of plugins can be 9 elements or longer and we have 3 redefinitions of the same list with only one plugin change each. The motivation to DRY this up is that our pipeline is already pretty long and over time when new cache layers are added, I don't want it to be so easy to forget to add it to the other plugin lists used by other steps.

**Describe the solution you'd like**
If the command plugins option also supported lists of lists of plugins as elements, we could do something like this simplified example:
```
docker-plugins: &docker-plugins
- docker-compose#v3.9.0: ~
- cache#v1.0.0:
keys: "v1-gems"
paths: "vendor/bundle"

asset-plugins: &asset-plugins
- cache#v1.0.0:
keys: "v1-assets"
paths: "public/assets"

steps:
- label: Do something with assets
command: ".buildkite/steps/asset_stuff"
plugins: *asset-plugins

- label: Do something with docker
command: ".buildkite/steps/docker_stuff"
plugins: *docker-plugins

- label: Do something with assets and docker
command: ".buildkite/steps/rspec"
plugins: [*asset-plugins, *docker-plugins]
```

The statement `[*asset-plugins, *docker-plugins]` produces a list with 2 elements both being lists of plugins. If the agent supported running a flatten operation on the plugins list, this would work.

**Describe alternatives you've considered**
I don't see any downsides to this solution, the plugin order is still clear as long as the flattening happens in order of list definition. It should also be a pretty small change, though might require a schema change in the backend to support validating the pipeline.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the agent's plugins option from YAML pipeline parsing through plugin validation; the request notes that backend schema validation may also need changes. Done means nested plugin lists are flattened in definition order and the provided YAML pattern is accepted.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, yaml
Domain
build-system, ci-cd
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.