GoogleContainerTools / GoogleContainerTools/skaffold
Add support for multi-arch OCI image indexes in Bazel builder
- Dominant language
- Go
- Stars
- 15.9k
- Forks
- 1.7k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 10
Description
I am working in a Bazel workspace that uses `rules_oci` to build image tarballs, and Skaffold to perform the build+deploy steps.
As I'm working on an M1 (and minikube) but deploying on a remote k8s cluster based on `linux-amd64`, the images I'm building are multi-arch.
When running `skaffold dev`, it fails with the following error:
```
skaffold dev -p local --port-forward
Generating tags...
- api -> api:v0.31.1-6-g1781fcd-dirty
Checking cache...
- api: Not found. Building
Starting build...
Found [minikube] context, using local docker daemon.
Building [api]...
Target platforms: [linux/arm64]
Loading:
Loading:
Loading: 0 packages loaded
Analyzing: target //backend/apps/api:image.tar (0 packages loaded, 0 targets configured)
INFO: Analyzed target //backend/apps/api:image.tar (46 packages loaded, 1470 targets configured).
INFO: Found 1 target...
[5 / 16] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Target //backend/apps/api:image.tar up-to-date:
bazel-bin/backend/apps/api/image.tar/tarball.tar
INFO: Elapsed time: 0.771s, Critical Path: 0.43s
INFO: 3 processes: 2 internal, 1 darwin-sandbox.
INFO: Build completed successfully, 3 total actions
Cleaning up...
- No resources found
Error: uninstall: Release not loaded: postgres: release: not found
Error: uninstall: Release not loaded: redis: release: not found
Error: uninstall: Release not loaded: jaeger: release: not found
Cleaning up resources encountered an error, will continue to clean up other resources.
build [api] failed: loading manifest from tarball failed: file manifest.json not found in tar
```
Here is an excerpt of the `BUILD.bazel` file, note how `format = "oci"` in `oci_tarball`:
BUILD.bazel
```starlark
go_binary(
name = "api",
embed = [":api_lib"],
visibility = ["//visibility:public"],
)
linux_amd64_binary = "api_linux-amd64".format(app_name)
linux_amd64_layer = "{}_layer".format(linux_amd64_binary)
linux_amd64_image = "{}_image".format(linux_amd64_binary)
go_cross_binary(
name = linux_amd64_binary,
platform = "@rules_go//go/toolchain:linux_amd64",
target = target,
)
pkg_tar(
name = linux_amd64_layer,
srcs = [linux_amd64_binary],
)
oci_image(
name = linux_amd64_image,
base = base,
entrypoint = ["/api"],
tars = [linux_amd64_layer],
)
linux_arm64_binary = "api_linux-arm64"
linux_arm64_layer = "{}_layer".format(linux_arm64_binary)
linux_arm64_image = "{}_image".format(linux_arm64_binary)
go_cross_binary(
name = linux_arm64_binary,
platform = "@rules_go//go/toolchain:linux_arm64",
target = target,
)
pkg_tar(
name = linux_arm64_layer,
srcs = [linux_arm64_binary],
)
oci_image(
name = linux_arm64_image,
base = base,
entrypoint = ["/api"],
tars = [linux_arm64_layer],
)
oci_image_index(
name = "image",
images = [
linux_amd64_image,
linux_arm64_image,
],
)
oci_tarball(
name = "image.tar",
format = "oci",
image = ":image",
repo_tags = ["api:latest"],
)
```
This is happening due to the following section: https://github.com/GoogleContainerTools/skaffold/blob/1255d6c4cdff405ef07da381dda1afcbbb62f153/pkg/skaffold/build/bazel/build.go#L93-L96
[`tarball.LoadManifest`](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/tarball#LoadManifest) looks for the `manifest.json` file in the tarball built by Bazel, but won't find it since OCI image indexes use an `index.json` file instead: https://github.com/google/go-containerregistry/blob/4fdaa32ee934cd178b6eb41b3096419a52ef426a/pkg/v1/tarball/image.go#L74-L79
`go-containerregistry` already exposes a package to load the `index.json` file: [`layout.ImageIndexFromPath`](https://pkg.go.dev/github.com/google/go-containerregistry@v0.17.0/pkg/v1/layout#ImageIndexFromPath)
`RepoTags` can be deducted using the stategy explained here: https://github.com/opencontainers/image-spec/issues/796
Contributor guide
Assessment
This issue has not been assessed yet.