Enhance Docker Build Script for Compatibility with Containerd Layer Store

Problem Description

The recent introduction of the containerd layer store in Depot-hosted runners exposed gaps in our understanding and configuration of Docker’s build process. The feature is working as designed, but it conflicts with our existing workflow when handling multi-architecture builds and provenance data.

Specifically:

• Single-architecture images inadvertently include manifest lists with an unexpected "unknown" platform due to Docker's default build provenance metadata.
• The legacy Docker layer store previously stripped this metadata, but the containerd store natively supports manifest lists and does not modify the image format.
• Modifying our scripts is necessary to fully support the containerd layer store while retaining provenance data and enabling multi-architecture builds.

Desired Solution

1. Update build_docker.sh to Use docker buildx:

   • Incorporate docker buildx build to handle multi-platform builds efficiently.
   • Ensure proper handling of provenance metadata and manifest merging when creating multi-arch images.

2. Integrate depot build as an Optional Workflow:

   • Add a flag to toggle between docker buildx and depot build for CI environments.
   • Use depot build for efficient multi-arch builds where possible, falling back to docker buildx for broader compatibility.

3. Enhance CI Workflow:

   • Default to docker buildx for local and dev workflows to avoid additional dependencies.
   • Enable depot build only in CI and set it up dynamically using a GitHub Action.

4. Optimize for Single-Arch Builds:

   • Modify the script to build single-architecture images efficiently (e.g., during dogfood development) while keeping multi-arch workflows intact in CI.

Next Steps

• Update scripts/build_docker.sh to support both docker buildx for local dev and depot build for ci.
• Test single-architecture and multi-architecture builds using both approaches to ensure functionality with the containerd layer store.

References

• Docker Buildx Documentation
• Docker Provenance Metadata
• containerd image store
• Multi-platform builds
• Slack Thread

This issue will help us fully leverage the containerd layer store's benefits while maintaining compatibility and development flexibility.

Beginne mit scripts/build_docker.sh und der darin referenzierten Dokumentation zu Docker Buildx, Provenance und Multi-Platform. Verfolge, wie das aktuelle Skript Builds für eine einzelne Architektur und Multi-Architecture handhabt, und vergleiche dann die angeforderten Workflows für docker buildx und den optionalen depot build. Als erledigt gilt die Aufgabe, wenn beide Workflows den containerd Layer Store handhaben und dabei Provenance sowie die CI-/lokale Kompatibilität erhalten.