Add the ability to mark an action as always needing to run
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the feature request:
In ChromeOS, we are using bazel to build packages. We also want to also use bazel to install these packages into an external (to bazel) sysroot. We previously used to have a `bazel run` action that would perform the installation of all packages in one go. This method required waiting for all packages to be built before installation could take place. Installing all the packages can take a while. To speed this up, we want to install each individual package as soon as possible. We ended up creating a [installation](https://chromium.googlesource.com/chromiumos/bazel/+/refs/heads/main/portage/build_defs/ebuild.bzl#801) build action that installs the package into the external sysroot as soon as the package's dependencies are installed into the sysroot, and the package is available.
The problem we are running into is that this action only runs once if none of the inputs change. It's possible for the external sysroot to be modified or even cleared between bazel invocations. We need a mechanism to force the action to always run.
This is what our current invocation looks like:
```
BOARD=amd64-generic bazel build @portage//target/virtual/target-os:installed
```
This will install all the OS packages into the sysroot at `/build/amd64-generic`.
Here is a simplified example of our current "install" rule:
```
ebuild_install_action(
name = "1.0-r2803_installed",
board = "amd64-generic",
package = ":1.0-r2803",
requires = [
"//internal/packages/stage2/target/board/chromiumos/chromeos-base/crosid:0.0.1-r227_installed",
"//internal/packages/stage2/target/board/chromiumos/dev-libs/nss:3.68.2-r5_installed",
"//internal/packages/stage2/target/board/chromiumos/sys-apps/coreboot-utils:0.0.1-r5575_installed",
"//internal/packages/stage2/target/board/chromiumos/sys-apps/flashrom:0.9.9-r1697_installed",
"//internal/packages/stage2/target/board/chromiumos/sys-apps/util-linux:2.38.1-r3_installed",
"//internal/packages/stage2/target/board/portage-stable/app-arch/libarchive:3.7.1_installed",
"//internal/packages/stage2/target/board/portage-stable/dev-libs/libzip:1.10.0-r1_installed",
"//internal/packages/stage2/target/board/portage-stable/dev-libs/openssl:3.2.1-r1_installed",
],
visibility = ["//:__subpackages__"],
)
```
The input to the rule is the package that we want installed, and the `_installed` action of all its dependencies.
One thought we had was to add a repository rule that analyzed the external sysroot and output a "digest" file that we consumed from the `install` action. This kind of works, but the repository rule is computed before the rule that modifies the sysroot, so on the next invocation the digest has changed and the install action runs again.
Put another way, we need a way to execute various `bazel run` actions according to the dep graph. At some point we will start running ebuild unit tests, and it would also be ideal if we could make the tests also run as part of this single invocation so that everything happens in parallel.
### Which category does this issue belong to?
Core
### What underlying problem are you trying to solve with this feature?
_No response_
### Which operating system are you running Bazel on?
_No response_
### What is the output of `bazel info release`?
_No response_
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
_No response_
### Have you found anything relevant by searching the web?
The interwebs suggest using `--workspace_status_command` and the version_file.
I naively tried the following patch, but the action only gets executed once:
```
diff --git a/portage/build_defs/ebuild.bzl b/portage/build_defs/ebuild.bzl
index 189655c8..6e3f7e8b 100644
--- a/portage/build_defs/ebuild.bzl
+++ b/portage/build_defs/ebuild.bzl
@@ -844,6 +844,10 @@ def _ebuild_install_action_impl(ctx):
args.add("-s")
args.add(dep[_EbuildInstalledInfo].checksum)
+ # We force the action to run every time by depending on the version_file
+ # which changes every invocation.
+ inputs.append(ctx.version_file)
+
ctx.actions.run(
executable = ctx.executable._action_wrapper,
inputs = inputs,
```
I suspect the version_file is special cased to not cause rebuilds every time? I didn't try adding the `--stamp` flag since I don't actually want a stamp.
Another option we could try is to have a repository rule that writes the [CACHE_BUST_DATE](https://chromium.googlesource.com/chromiumos/bazel/+/refs/heads/main/workspace_root/alchemy/tools/bazel#66) to a file and have the `install` action depend on that. I tried using `--action_env=$CACHE_BUST_DATE`, but I wasn't able to get that to work.
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
Start with the installation rule in portage/build_defs/ebuild.bzl and the reported --workspace_status_command and version_file approaches. No Bazel source entry point or test is named, so first locate the action-cache behavior governing repeated execution. Done means a supported mechanism reruns the dependency-graph action after the external sysroot changes or is cleared.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100