bazel-contrib / bazel-contrib/rules_oci

`oci_load` is incompatible with cross architecture remote execution.

Open
#769 0 comments 0 reactions 1 assignee Claimed by @thesayyn View on GitHub
need: investigation
Dominant language
Starlark
Stars
423
Forks
213
Avg merge
53m
Merged PRs (30d)
1

Description

Hi,
Since `rules_oci` started using tar toolchain, the `oci_load` has a subtle issue with remote execution where `host_platform != exec_platform`

Consider the following example:

```
bazel run //:my_image_tar.amd64 --config=remote --extra_execution_platforms=linux_x86_64
```

The produced load script will select `exec_compatible_with` tar toolchain - `linux_x86_64` and then will be executed on host, for instance `darwin_arm64`.

I've tried to workaround this by

```
def host_exec_compatible_rule(rule, name, **kwargs):
rule(
name = name,
exec_compatible_with = [
host_cpu_constraint(),
host_os_constraint(),
],
**kwargs
)
```

However, such approach makes incorrect toolchain resolution for [building tarball itself](https://github.com/bazel-contrib/rules_oci/blob/7de244c3cdc78fb002b1b01b2806a1dc6bd5a1aa/oci/private/load.bzl#L193). The toolchains are chosen for host platform, while the rule is executed on linux_x86_64.

Similar, but not 100% same issue: https://github.com/bazelbuild/bazel/issues/22848 . The workaround with `local_only_rule` works in this case as well, but it defeats the purpose of remote execution a bit and here there is no reason to use `no-remote-exec` tag.

That said, I think the problem may be very hard to solve:
* executables (load script here) can be both:
* executed directly with `bazel run` - in this case toolchains should be selected for host
* in principle be used as dependency to a build rule - in this case, the build rule would be executed remotely, so toolchains should be selected for execution platform.
* Switching to `target_compatible_with` toolchain for the load script is probably infeasible. It would solve `bazel run` problem, but would break build rules that depend on such script.

**EDIT** I believe that if one needs to depend on the script in another rule, they can use `cfg = exec`. This makes me think that the load script should depend on tar toolchain that is `target_compatible_with`

Looking forward to your thoughts.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.