kubernetes: build with an initContainer or a Job
- Dominant language
- Go
- Stars
- 300
- Forks
- 64
- Avg merge
- 20m
- Merged PRs (30d)
- 1
Description
## Context
Currently, envbuilder runs at the start up of a workspace, exposing elements of the buildtime to the runtime and vice-versa:
* Build secrets (e.g. dockerconfig)
* Environment variables (#91)
* Mounts (#187)
* Privileges (#181)
* Container layers are downloaded in each container rather than on the nodes
* ... (feel free to grow the list)
## Proposal 1: initContainer
1. Envbuilder would build the image as an [initContainer](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) and push it to a container registry
2. The main container would pull and run the image (todo: validate that the pod can be created without the image existing yet)
This would require to generate/know the image reference ahead of time.
## Proposal 2: Kubernetes Job
Entire decoupling of buildtime and runtime:
1. Envbuilder runs as Kubernetes `Job` to build and push the container image
2. It writes a `ConfigMap` with the reference of the built image
3. Terraform [waits for completion](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/job#wait_for_completion) of the `Job`
4. Terraform reads the `ConfigMap` with [`kubernetes_config_map`](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/data-sources/config_map) datasource ([explicitly depending on](https://developer.hashicorp.com/terraform/language/meta-arguments/depends_on) the `Job` creation)
5. The image reference from the `ConfigMap` is then used to create a `Deployment`
6. A short [`ttl_seconds_after_finished`](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/job#ttl_seconds_after_finished) would allow clean up of the Job for it to be recreated on the next `terraform apply`
The `ConfigMap` could be used to share of information between `envbuilder` and Terraform (#121), like the volumes defined in the `devcontainer.json` (#220)
Detail to consider: I believe the Coder server starts streaming the logs from the deployment after the `terraform apply` has finished, it would need to be able to do it for the `Job` while the apply is running to expose the build logs to the user.
Is it something that has been thought of/done but not documented yet?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.