GoogleContainerTools / GoogleContainerTools/skaffold

Add support for multi-arch OCI image indexes in Bazel builder

Open
#9,217 2 comments 0 reactions 0 assignees View on GitHub
build/bazel priority/p2
Dominant language
Go
Stars
15.9k
Forks
1.7k
Avg merge
3d 9h
Merged PRs (30d)
10

Description

I am working in a Bazel workspace that uses `rules_oci` to build image tarballs, and Skaffold to perform the build+deploy steps.

As I'm working on an M1 (and minikube) but deploying on a remote k8s cluster based on `linux-amd64`, the images I'm building are multi-arch.

When running `skaffold dev`, it fails with the following error:
```
skaffold dev -p local --port-forward
Generating tags...
- api -> api:v0.31.1-6-g1781fcd-dirty
Checking cache...
- api: Not found. Building
Starting build...
Found [minikube] context, using local docker daemon.
Building [api]...
Target platforms: [linux/arm64]
Loading:
Loading:
Loading: 0 packages loaded
Analyzing: target //backend/apps/api:image.tar (0 packages loaded, 0 targets configured)
INFO: Analyzed target //backend/apps/api:image.tar (46 packages loaded, 1470 targets configured).
INFO: Found 1 target...
[5 / 16] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Target //backend/apps/api:image.tar up-to-date:
bazel-bin/backend/apps/api/image.tar/tarball.tar
INFO: Elapsed time: 0.771s, Critical Path: 0.43s
INFO: 3 processes: 2 internal, 1 darwin-sandbox.
INFO: Build completed successfully, 3 total actions
Cleaning up...
- No resources found
Error: uninstall: Release not loaded: postgres: release: not found
Error: uninstall: Release not loaded: redis: release: not found
Error: uninstall: Release not loaded: jaeger: release: not found
Cleaning up resources encountered an error, will continue to clean up other resources.
build [api] failed: loading manifest from tarball failed: file manifest.json not found in tar
```

Here is an excerpt of the `BUILD.bazel` file, note how `format = "oci"` in `oci_tarball`:

BUILD.bazel

```starlark
go_binary(
name = "api",
embed = [":api_lib"],
visibility = ["//visibility:public"],
)

linux_amd64_binary = "api_linux-amd64".format(app_name)
linux_amd64_layer = "{}_layer".format(linux_amd64_binary)
linux_amd64_image = "{}_image".format(linux_amd64_binary)

go_cross_binary(
name = linux_amd64_binary,
platform = "@rules_go//go/toolchain:linux_amd64",
target = target,
)

pkg_tar(
name = linux_amd64_layer,
srcs = [linux_amd64_binary],
)

oci_image(
name = linux_amd64_image,
base = base,
entrypoint = ["/api"],
tars = [linux_amd64_layer],
)

linux_arm64_binary = "api_linux-arm64"
linux_arm64_layer = "{}_layer".format(linux_arm64_binary)
linux_arm64_image = "{}_image".format(linux_arm64_binary)

go_cross_binary(
name = linux_arm64_binary,
platform = "@rules_go//go/toolchain:linux_arm64",
target = target,
)

pkg_tar(
name = linux_arm64_layer,
srcs = [linux_arm64_binary],
)

oci_image(
name = linux_arm64_image,
base = base,
entrypoint = ["/api"],
tars = [linux_arm64_layer],
)

oci_image_index(
name = "image",
images = [
linux_amd64_image,
linux_arm64_image,
],
)

oci_tarball(
name = "image.tar",
format = "oci",
image = ":image",
repo_tags = ["api:latest"],
)
```

This is happening due to the following section: https://github.com/GoogleContainerTools/skaffold/blob/1255d6c4cdff405ef07da381dda1afcbbb62f153/pkg/skaffold/build/bazel/build.go#L93-L96

[`tarball.LoadManifest`](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/tarball#LoadManifest) looks for the `manifest.json` file in the tarball built by Bazel, but won't find it since OCI image indexes use an `index.json` file instead: https://github.com/google/go-containerregistry/blob/4fdaa32ee934cd178b6eb41b3096419a52ef426a/pkg/v1/tarball/image.go#L74-L79

`go-containerregistry` already exposes a package to load the `index.json` file: [`layout.ImageIndexFromPath`](https://pkg.go.dev/github.com/google/go-containerregistry@v0.17.0/pkg/v1/layout#ImageIndexFromPath)

`RepoTags` can be deducted using the stategy explained here: https://github.com/opencontainers/image-spec/issues/796

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.