buildkite / buildkite/agent-stack-k8s

[FEATURE] Support exposing pod template choices for pipeline steps

Open
#854 3 comments 2 reactions 1 assignee Claimed by @zhming0 View on GitHub
enhancement
Dominant language
Go
Stars
110
Forks
57
Avg merge
3d 8h
Merged PRs (30d)
5

Description

## Is your feature request related to a problem? Please describe.

I'd like to provide step authors the ability to select a pod template while keeping `prohibit-kubernetes-plugin=true`, ideally in a composable and relatively secure way.

The core problem is that users can't specify the "shape" of their pods unless operators give them full reign via `podSpecPatch`.

Right now, we can specify `resource-classes`, which lets users select from some pre-populated requests-limits pairs. This is great! However, I have users who want to use docker. To support this, I'm running a docker-in-docker sidecar (DinD) by setting an agent-level `pod-spec-patch`.

However, since the `pod-spec-patch` is static, the only way I've found to support giving DinD different resources is by running multiple queues+controllers+helm installs. Some docker builds are differently resource-intensive, so I have basically replicated the `resource-classes` configuration in the form of queues with different DinD resource specs.

Furthermore, not all of my job steps use DinD, so I need to have additional analagous queues that don't stick the container into the pod.

## Describe the solution you'd like

I'm not married to a specific implementation, but I think something like this might make it easier for me to manage:

### Support `template-classes` or something similar

Something like this would let me specify various profiles of the "shape" of the generated job pod:

```yaml
# values.yaml
config:
template-classes:
normal-job-runner: default-podtemplate-name
docker-job-runner: docker-podtemplate-name
default-template-class-name: normal-job-runner

resource-classes:
small:
resource:
requests:
cpu: "1"
memory: 1Gi
limits:
cpu: "1"
memory: 1Gi
large:
resource:
requests:
cpu: "8"
memory: 32Gi
limits:
cpu: "8"
memory: 32Gi

```

Here, `template-classes` is an object:

- key: some slug that configures the name of a "template class" that can be used in a pipeline step specification
- value: name of a PodTemplate k8s resource that will be used to generate the build's job pod

A step specification could then optionally select one of the template classes:

```yaml
# pipeline.yaml
steps:
- label: "big build"
command: "make build"
agents:
resource_class: "large"

steps:
- label: "docker build"
command: "docker build ."
agents:
resource_class: "small" # still nice to configure container-0
template_class: "docker-job-runner" # <-- NEW: the job is run with a specific PodTemplate that adds the DinD container to the pod
```

## Describe alternatives you've considered

There are a lot of potential solutions to this problem. Taken to its logical conclusion, the most generalized solution is probably akin to `kustomize`, where you can edit literally anything. However, that might be overdoing it and not aligned with the focused style of buildkite that I appreciate.

I think being able to specify a PodTemplate is a reasonable middleground where operators can provide users with composable functionality. We can also be sure that a PodTemplate accessible through the API has passed some validation.

One failure mode to consider is when the step-specified pod template doesn't exist, but this failure type is analagous to a resource_class not existing.

In any case, managing N pod templates feels cleaner to me than managing N helm deployments, and with the `pod-spec-patch` escape hatch, we unlock even more configurability.

## Additional context

In the general case, the current design seems to require a lot of queues:

```
number of queues = (number of configurable dimensions) * (avg number of values in each dimension)
```

As soon as I want another optional sidecar with a bit of configuration, I'm facing a combinatorial explosion of queues+controllers. In a sense, the proposed solution doesn't solve this general case, but it would get me pretty far.

Side note 1: we can use `podTemplate` and `podSpecPatch` from the step spec, but on the controller/helm chart, we cannot specify `pod-template` (only `pod-spec-patch`)! This would be nice to change. This means I have big piles of almost-equivalent yaml for each queue.

Side note 2: it would also be nice to be able to specify resources for containers other than `container-0`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.