buildkite / buildkite/agent-stack-k8s
[FEATURE] Support exposing pod template choices for pipeline steps
- Dominant language
- Go
- Stars
- 110
- Forks
- 57
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 5
Description
## Is your feature request related to a problem? Please describe.
I'd like to provide step authors the ability to select a pod template while keeping `prohibit-kubernetes-plugin=true`, ideally in a composable and relatively secure way.
The core problem is that users can't specify the "shape" of their pods unless operators give them full reign via `podSpecPatch`.
Right now, we can specify `resource-classes`, which lets users select from some pre-populated requests-limits pairs. This is great! However, I have users who want to use docker. To support this, I'm running a docker-in-docker sidecar (DinD) by setting an agent-level `pod-spec-patch`.
However, since the `pod-spec-patch` is static, the only way I've found to support giving DinD different resources is by running multiple queues+controllers+helm installs. Some docker builds are differently resource-intensive, so I have basically replicated the `resource-classes` configuration in the form of queues with different DinD resource specs.
Furthermore, not all of my job steps use DinD, so I need to have additional analagous queues that don't stick the container into the pod.
## Describe the solution you'd like
I'm not married to a specific implementation, but I think something like this might make it easier for me to manage:
### Support `template-classes` or something similar
Something like this would let me specify various profiles of the "shape" of the generated job pod:
```yaml
# values.yaml
config:
template-classes:
normal-job-runner: default-podtemplate-name
docker-job-runner: docker-podtemplate-name
default-template-class-name: normal-job-runner
resource-classes:
small:
resource:
requests:
cpu: "1"
memory: 1Gi
limits:
cpu: "1"
memory: 1Gi
large:
resource:
requests:
cpu: "8"
memory: 32Gi
limits:
cpu: "8"
memory: 32Gi
```
Here, `template-classes` is an object:
- key: some slug that configures the name of a "template class" that can be used in a pipeline step specification
- value: name of a PodTemplate k8s resource that will be used to generate the build's job pod
A step specification could then optionally select one of the template classes:
```yaml
# pipeline.yaml
steps:
- label: "big build"
command: "make build"
agents:
resource_class: "large"
steps:
- label: "docker build"
command: "docker build ."
agents:
resource_class: "small" # still nice to configure container-0
template_class: "docker-job-runner" # <-- NEW: the job is run with a specific PodTemplate that adds the DinD container to the pod
```
## Describe alternatives you've considered
There are a lot of potential solutions to this problem. Taken to its logical conclusion, the most generalized solution is probably akin to `kustomize`, where you can edit literally anything. However, that might be overdoing it and not aligned with the focused style of buildkite that I appreciate.
I think being able to specify a PodTemplate is a reasonable middleground where operators can provide users with composable functionality. We can also be sure that a PodTemplate accessible through the API has passed some validation.
One failure mode to consider is when the step-specified pod template doesn't exist, but this failure type is analagous to a resource_class not existing.
In any case, managing N pod templates feels cleaner to me than managing N helm deployments, and with the `pod-spec-patch` escape hatch, we unlock even more configurability.
## Additional context
In the general case, the current design seems to require a lot of queues:
```
number of queues = (number of configurable dimensions) * (avg number of values in each dimension)
```
As soon as I want another optional sidecar with a bit of configuration, I'm facing a combinatorial explosion of queues+controllers. In a sense, the proposed solution doesn't solve this general case, but it would get me pretty far.
Side note 1: we can use `podTemplate` and `podSpecPatch` from the step spec, but on the controller/helm chart, we cannot specify `pod-template` (only `pod-spec-patch`)! This would be nice to change. This means I have big piles of almost-equivalent yaml for each queue.
Side note 2: it would also be nice to be able to specify resources for containers other than `container-0`
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.