Server-side container image builds

As a data scientist / ML engineer, I do not want to go through the overhead of downloading all sorts of dependencies and uploading them back to the internet as a container image, before I can sub my job for execution (since I am on a limited-bandwidth connection).

Instead, I want these container images to be built on a build server, according to the image specification given as part of my job metadata.

This approach yields additional benefits:

- Improved caching (esp. in teams), since images don't need to be built multiple times on each team members' local machine
- No architecture gap (e.g., macOS/Linux, ARM/x64) between clients and compute server
- No need for data scientists to install container tooling on their machine
- Potential for security measures on the build server (e.g., vulnerability scanning, image signing, SBOM, ...)

---

# High-level Design

## Overview

- Server offers an asynchronous `POST /builds` API endpoint that takes the following information
- Image spec / Dockerfile
- Build context archive (as .tar.gz)
- Image tag
- Target platform (optionally)
- Actual image building happens as a background task, the client may poll the status through a `GET /builds/` API endpoint
- Build logs can be made available to the client through a `GET /builds//logs` endpoint (both historic and streaming logs)

## Server-side

- Need to provide the above APIs
- Actual image build process should be decoupled from the API
- Simplest approach: FastAPI `BackgroundTasks`
- Could be a completely separate service
- Decide whether to use Docker daemon (as we have before) or Buildkit (more flexible, especially when it comes to the output options and separation between multiple projects) to build the image
- Image publishing needs to happen here (see open questions below)

## Client-side

- Putting together the build context in a similar fashion to the Docker CLI (unfortunately, this functionality is not exposed to the user)
- this means respecting the ignore patterns given in a `.dockerignore` or deriving sensible defaults (e.g., from `.gitignore`)
- Removing all image build functionality and replacing it with API calls
- Moving all image spec functionality into the backend (basically the entire `jobq.assembler` package)

## Additional Considerations

### Security

- Extracting a user-supplied build context on the build server exposes a DoS risk due to disk space exhaustion (c.f. [zipbomb](https://en.wikipedia.org/wiki/Zip_bomb))
- On Linux [cgroups](https://man7.org/linux/man-pages/man7/cgroups.7.html) might come in handy to set resource quotas for the extraction process.
- Cross-team layer sharing could leak information (but it also desirable to speed up build process)
- Mitigation might involve separate `buildx` builders or `buildkitd` instances per jobq team/project (#130)

### Correctness

- Since the assembly process for the build context will have to re-implemented, we should make sure that we closely mirror the behavior of `docker build`.
- Choosing sensible defaults for the context ignore patterns in the absence of a `.dockerignore` file

## Open Questions

- Control flow between the `POST /jobs` and image build API?
- Does the client need to orchestrate the build process, or will the job submission endpoint change to take in all the required data (i.e., build context and image build spec) and abstract the image build process internally?
- When the client needs to orchestrate the API calls, it also implies that it will have to periodically poll the image build status before calling the job submission (ergo, it needs to block the user indefinitely).
- Combining image builds with job submission probably means extending the job state machine, with extra lifecycle states for the image build process (so that the submission endpoint can return a valid job ID right away while the job has not been submitted yet to the cluster). Problem: These are not coupled to Kueue since the job has not yet been submitted to the cluster. This directly raises the question of persistent state storage in the backend (see below).
- Storing persistent state in the backend (see also #130): Eventually, the backend will have to persist some state (e.g., build job status, or even a full database of jobs and related entities). Do we bite the bullet now and design a persistent state management component, or can we get away with something ephemeral for now
- E.g., if the client orchestrates the build process, it could be okay if the server returns an HTTP 404 error for the build status if it was restarted during the image build - the client would have to resubmit the image build request in this case (Q: does it need to keep the build context around until then, or is it acceptable to just re-create the archive, possibly with modified contents?).
- **Image naming**: Who decides how images are named and tagged (user, CLI, API)?
- **Image destination**: Where will images be published? Should we bundle an image registry with the default setup or delegate to external ones (which might complicate the topic of passing credentials)?
- Should image build resource usage be monitored and used for accounting purposes?
- Should it be possible to set resource limits for image builds, similar to the workflow execution? If yes, could these be unified (i.e., implement image build tasks as Kueue workloads)?

### Client-based Orchestration

```mermaid
sequenceDiagram
participant Job API
actor CLI as Jobq User
participant Build API
participant Background Tasks
participant Builder

CLI->>Build API: POST /build
activate CLI
activate Build API

Build API -) Background Tasks: Trigger image build
Background Tasks ->> Builder: Build image
activate Builder
Build API-->>CLI: HTTP 202, build_id, image_ref
deactivate Build API

par
loop status != "completed"
CLI ->> Build API: GET /builds/
activate Build API
Build API -->> CLI: Build status
deactivate Build API
end
and
Builder -->> Background Tasks: image_ref
deactivate Builder
Note right of Background Tasks: Publish image here?
end

CLI->>Job API: POST /jobs (image_ref)
activate Job API
Job API -->> CLI: HTTP 200, job_id
deactivate Job API

deactivate CLI
```