microsoft / microsoft/mssql-rs

Retain a rollback tag for pre-built Kerberos test images instead of only overwriting `latest`

Open
#180 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
53
Forks
14
Avg merge
1d 15h
Merged PRs (30d)
137

Description

Problem statement

The Kerberos test images published by .pipeline/sync-container-images.yml are tagged latest and nothing else. The weekly schedule (cron: "0 3 * * 1", always: true) rebuilds and force-pushes those same tags every Monday, so the previous known-good image is destroyed on every run.

Affected tags (11 total):

kerberos/samba-dc:latest
kerberos/mssql-ad:latest
kerberos/client:{alpine318,alpine319,alpine320,alpine321,debian,ubuntu22,ubuntu24,rhel9,oracle9}-latest

Both PR and CI Kerberos stages consume these via .pipeline/templates/kerberos-test-template.yml, which defaults imageTag to latest and pulls only — there is no local build fallback, by design. That makes the images a hard dependency of the Kerberos stage.

The failure mode this creates: if a weekly run publishes a broken-but-pushable image (bad base image, upstream package regression, partially-applied change), latest is already overwritten by the time anyone notices. There is nothing to roll back to. Every Kerberos run on every PR fails until someone re-runs the sync pipeline from a known-good commit and waits for a full rebuild.

Note this is specifically about a bad image, not a missing one. A missing image fails at pull time and is obvious. A bad image fails later, during test execution, and looks like a product regression.

Proposed solution

Retain at least one prior generation so there is something to point at when latest is bad.

Sketch, not prescriptive:

  • Before pushing a new latest, retag the current one as previous (or latest-1).
  • Alternatively, push an immutable stamped tag alongside latest on every run — date (2026-08-05), build ID, or source SHA — and keep latest as a moving pointer. This generalises better than a single previous slot and gives more than one generation of history.
  • Consumers keep defaulting to latest; the stamped/previous tag is what you set imageTag to when you need to pin or roll back.

Rolling back then becomes a parameter change on the pipeline run rather than a rebuild.

Open questions

Should the consumer fall back automatically? I would lean no, and it should be a deliberate decision rather than a default. Automatically retrying a failed pull against previous gives a green pipeline running week-old images, which is the same "green but wrong" hazard that led us to reject a local-build fallback in #171. Explicit rollback (someone sets imageTag) keeps the failure visible while still making recovery fast. Worth confirming.

Retention. GHCR will accumulate stamped tags indefinitely. Needs either a retention policy or a deliberate decision to let them accumulate — these images are 1.5–3 GB each for the clients.

Client tag scheme. The clients currently encode the distro in the tag (kerberos/client:ubuntu22-latest) rather than the repository (kerberos/client-ubuntu22:latest). That works for latest, but it means any versioning scheme has to compose two dimensions into one tag string (ubuntu22-2026-08-05), and it makes per-distro digest pinning awkward. If we are changing the tagging scheme anyway, this is the moment to consider moving the distro into the repo name — it costs a republish either way, and the cost only grows.

Additional context

Context from #171 (consumer side) and #163 (producer side).

Relevant code:

  • .pipeline/sync-container-images.ymlBuildKerberosImages job builds and pushes the :latest tags to ACR; Mirror_To_GHCR copies them to GHCR from the list in the images.txt heredoc.
  • .pipeline/templates/kerberos-test-template.ymlimageRegistry / imageTag parameters; the pull step exits non-zero rather than falling back.

Worth noting the tags are currently static in practice — the weekly schedule only republishes from main, and the images were last built from the #163 branch run. This becomes an active risk the first Monday after #163's merge, when the scheduled job starts overwriting them on a cadence.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read .pipeline/sync-container-images.yml, especially BuildKerberosImages, Mirror_To_GHCR, and the images.txt heredoc, then inspect .pipeline/templates/kerberos-test-template.yml. Trace how imageTag is passed and verify the chosen retention approach works for all listed images. Done means a prior image remains recoverable without changing the default consumer behavior, with retention and tag-format decisions resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
ci-cd, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.