dotnet / dotnet/docker-tools

Rework container image rebuilds

Open
#2,166 1 comment 0 reactions 1 assignee Claimed by @lbussell View on GitHub
area-infrastructure
Dominant language
C#
Stars
181
Forks
67
Avg merge
2d 15h
Merged PRs (30d)
10

Description

## Current state

```mermaid
graph TD

subgraph docker-tools repo
check-base-images["check base images pipeline \n (multiple repos)"]
subscriptions[subscriptions.json]
end

subgraph consuming["consuming repo"]
build-pipeline[build-pipeline]
end

subgraph versions["versions repo"]
image-info[image-info.json]
end

subgraph acr
public["public images"]
end

check-base-images -->|reads| subscriptions
check-base-images -->|"reads (many)"| image-info
check-base-images -->|"queues (many)"| build-pipeline
subscriptions -->|"references (many)"| consuming
build-pipeline <-->|updates and reads| image-info
build-pipeline -->|pushes images| public
```

### Problems with current setup

- Requires a separate repo to store `image-info.json` metadata file. Git is in the critical path for builds.
- Only the docker-tools repo can queue base image re-bulids, and the docker-tools repo must know about all other consuming repos in order to queue their builds.
- The docker-tools repo and the consuming repos can go out-of-sync. For example, if the manifest.json or image-info formats change, then the docker-tools check-base-images pipeline can fail until all consuming repos have adopted the updated image-builder version.

## Desired state

**Key ideas**:
- Consuming repos should own their own base image update pipeline + schedule.
- The docker-tools repo should provide sane defaults/pipeline templates for base image re-builds.
- Base image re-builds should work both in public and internal scenarios without extra work.
- There should not be a dependency on external repos or storage locations for the critical base image re-build path. `image-info.json`, if it still needs to exist, should not be a source of truth. It should be computed separately, strictly downstream of image build pipelines.

```mermaid
graph TD

subgraph consuming[consuming repo]
check-base-images["check base images pipeline \n (one repo)"]
build-pipeline[build pipeline]
end

subgraph acr
public["public images \n with labels/annotations"]
end

public -->|reads| check-base-images
public -->|reads| build-pipeline
check-base-images -->|queues| build-pipeline
build-pipeline -->|pushes images| public
```

## Future work

Here's the work that I see required to reach this desired state:

- [ ] #1602
- [ ] Attach metadata labels to images during build - #2163
- [ ] Make rebuild decisions repo-local
- [ ] Implement a new version of the `getStaleImages` command which reads these labels to determine which images are out of date.
- [ ] Implement a new pipeline which runs the new command and queues a build for stale images.
- [ ] Backwards compatibility with existing image-info.json publishing
- ImageBuilder should still synthesize image-info.json and write it to disk, at least for the time being.
- For backwards compatibility, we need to provide some kind paved path for publishing image-info.json to a location like a git repo, OCI artifact, or storage account. This could be in the form of a pipeline extension point/parameter, a new imagebuilder command, a totally new pipeline/template, or some combination of those.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.