Support flattening lists in step plugins list
- Dominant language
- Go
- Stars
- 1.1k
- Forks
- 378
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 74
Description
**Is your feature request related to a problem? Please describe.**
In an effort to keep our pipeline steps more concise we're using YAML anchors and aliases to define the plugins for our steps at the top level and we reuse the lists in multiple places. This works well but we have some steps that need to fetch artifacts or add additional plugins. Since YAML doesn't support merging lists with anchors like it does for dictionaries, we need to redefine the list of plugins for these other use cases. Because of how our caches are layered, our lists of plugins can be 9 elements or longer and we have 3 redefinitions of the same list with only one plugin change each. The motivation to DRY this up is that our pipeline is already pretty long and over time when new cache layers are added, I don't want it to be so easy to forget to add it to the other plugin lists used by other steps.
**Describe the solution you'd like**
If the command plugins option also supported lists of lists of plugins as elements, we could do something like this simplified example:
```
docker-plugins: &docker-plugins
- docker-compose#v3.9.0: ~
- cache#v1.0.0:
keys: "v1-gems"
paths: "vendor/bundle"
asset-plugins: &asset-plugins
- cache#v1.0.0:
keys: "v1-assets"
paths: "public/assets"
steps:
- label: Do something with assets
command: ".buildkite/steps/asset_stuff"
plugins: *asset-plugins
- label: Do something with docker
command: ".buildkite/steps/docker_stuff"
plugins: *docker-plugins
- label: Do something with assets and docker
command: ".buildkite/steps/rspec"
plugins: [*asset-plugins, *docker-plugins]
```
The statement `[*asset-plugins, *docker-plugins]` produces a list with 2 elements both being lists of plugins. If the agent supported running a flatten operation on the plugins list, this would work.
**Describe alternatives you've considered**
I don't see any downsides to this solution, the plugin order is still clear as long as the flattening happens in order of list definition. It should also be a pretty small change, though might require a schema change in the backend to support validating the pipeline.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
Start by tracing the agent's plugins option from YAML pipeline parsing through plugin validation; the request notes that backend schema validation may also need changes. Done means nested plugin lists are flattened in definition order and the provided YAML pattern is accepted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, yaml
- Domain
- build-system, ci-cd
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100