buildkite / buildkite/agent

Support default convention for dynamic pipeline generation

Open
#3,118 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.1k
Forks
378
Avg merge
2d 6h
Merged PRs (30d)
74

Description

**Is your feature request related to a problem? Please describe.**
I want to generate a pipeline dynamically but don't want to change the default pipeline defined in the Buildkite UI and don't want extra build steps for every build just to generate/upload dynamic pipelines.

**Describe the solution you'd like**
We show an example in the docs of:
```
$ ./script/dynamic_step_generator | buildkite-agent pipeline upload
```

My request is to have a conventional default for that behavior rather than requiring a custom pipeline to invoke it.

Add a default "dynamic pipeline generator script" to the [list of pipeline files](https://buildkite.com/docs/agent/v3/cli-pipeline#uploading-pipelines-description) that the buildkite-agent automatically detects.

Specific name doesn't matter but `buildkite.sh` / `.buildkite/pipeline.sh` seems reasonable.

When `buildkite-agent pipeline upload` is run and a `buildkite.sh` or `.buildkite/pipeline.sh` file exists and is executable, execute it and take the output as a pipeline.

**Describe alternatives you've considered**
* Living with having one CI job step to "upload pipeline" and then another immediately after to "generate and upload dynamic pipeline"
* Editing each project in the Buildkite UI where we use dynamic pipelines to have a different default pipeline. This doesn't feel great for us as we really want the pipeline logic to fully live in the git repo.

**Additional context**
N/A

Contributor guide

Open the contributing guide

Research direction

Start at the `buildkite-agent pipeline upload` entry point and read the pipeline-uploading documentation linked in the issue. Investigate how `buildkite.sh` and `.buildkite/pipeline.sh` could be detected and invoked only when executable; done means their output is accepted as the pipeline without requiring an extra configured build step.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ci-cd, cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.