coder / coder/envbuilder

kubernetes: build with an initContainer or a Job

Open
#269 2 comments 3 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
300
Forks
64
Avg merge
20m
Merged PRs (30d)
1

Description

## Context

Currently, envbuilder runs at the start up of a workspace, exposing elements of the buildtime to the runtime and vice-versa:

* Build secrets (e.g. dockerconfig)
* Environment variables (#91)
* Mounts (#187)
* Privileges (#181)
* Container layers are downloaded in each container rather than on the nodes
* ... (feel free to grow the list)

## Proposal 1: initContainer

1. Envbuilder would build the image as an [initContainer](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) and push it to a container registry
2. The main container would pull and run the image (todo: validate that the pod can be created without the image existing yet)

This would require to generate/know the image reference ahead of time.

## Proposal 2: Kubernetes Job

Entire decoupling of buildtime and runtime:

1. Envbuilder runs as Kubernetes `Job` to build and push the container image
2. It writes a `ConfigMap` with the reference of the built image
3. Terraform [waits for completion](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/job#wait_for_completion) of the `Job`
4. Terraform reads the `ConfigMap` with [`kubernetes_config_map`](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/data-sources/config_map) datasource ([explicitly depending on](https://developer.hashicorp.com/terraform/language/meta-arguments/depends_on) the `Job` creation)
5. The image reference from the `ConfigMap` is then used to create a `Deployment`
6. A short [`ttl_seconds_after_finished`](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/job#ttl_seconds_after_finished) would allow clean up of the Job for it to be recreated on the next `terraform apply`

The `ConfigMap` could be used to share of information between `envbuilder` and Terraform (#121), like the volumes defined in the `devcontainer.json` (#220)

Detail to consider: I believe the Coder server starts streaming the logs from the deployment after the `terraform apply` has finished, it would need to be able to do it for the `Job` while the apply is running to expose the build logs to the user.

Is it something that has been thought of/done but not documented yet?

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by reviewing the current Kubernetes deployment and Terraform resources, then trace how envbuilder builds during workspace startup; done requires choosing and implementing one proposal, validating the image-reference handoff, and addressing build-log streaming.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, terraform
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.